AML Quality Assurance & Improvement 2 — Questions and Answers
Question 1: Which cross-validation strategy is most appropriate when your dataset has significant temporal ordering?
- Stratified k-fold
- Leave-one-out CV
- Time-series split (walk-forward) (Correct answer)
- Shuffle-split CV
Correct answer: Time-series split (walk-forward)
Time-series split (walk-forward validation) prevents data leakage by always training on past data and validating on future data.
Question 2: A model achieves 99% accuracy on an imbalanced dataset where 99% of samples are class 0. What is the most important additional metric to evaluate?
- Top-k accuracy
- Matthews Correlation Coefficient (MCC) (Correct answer)
- Log loss
- Adjusted R-squared
Correct answer: Matthews Correlation Coefficient (MCC)
MCC accounts for all four confusion matrix values and provides a reliable metric even when classes are highly imbalanced.
Question 3: What does a calibration curve (reliability diagram) measure in a classifier?
- Speed of convergence during training
- Alignment between predicted probabilities and actual outcome frequencies (Correct answer)
- Variance of predictions across bootstrap samples
- Feature importance ranking stability
Correct answer: Alignment between predicted probabilities and actual outcome frequencies
A calibration curve plots predicted probabilities against actual frequencies to show whether a model's confidence scores are trustworthy.
Question 4: You observe that validation loss stops improving after epoch 20 but training loss keeps decreasing. The correct QA response is to:
- Increase model capacity and retrain
- Apply early stopping at epoch 20 and regularize the model (Correct answer)
- Reduce the learning rate and continue training indefinitely
- Switch to a different optimizer
Correct answer: Apply early stopping at epoch 20 and regularize the model
The divergence between training and validation loss is a clear overfitting signal; early stopping and regularization directly address it.
Question 5: Which technique quantifies uncertainty by training multiple models on bootstrap samples of the training data?
- Monte Carlo Dropout
- Deep ensembles
- Bootstrap aggregating (Bagging) (Correct answer)
- Platt scaling
Correct answer: Bootstrap aggregating (Bagging)
Bagging trains multiple models on bootstrapped datasets; the variance of their predictions estimates model uncertainty.
Question 6: During error analysis, you find that 80% of misclassifications occur on one specific data slice. The best QA action is to:
- Remove the problematic slice from evaluation
- Upsample that slice and retrain with targeted augmentation (Correct answer)
- Accept the error rate as statistically inevitable
- Reduce the decision threshold uniformly
Correct answer: Upsample that slice and retrain with targeted augmentation
Targeted upsampling and augmentation of the underperforming slice directly addresses the root cause of slice-specific failures.
Question 7: The Brier Score is used to evaluate:
- Ranking quality of a retrieval model
- Calibration and sharpness of probabilistic forecasts (Correct answer)
- Feature redundancy in high-dimensional spaces
- Clustering cohesion and separation
Correct answer: Calibration and sharpness of probabilistic forecasts
The Brier Score is the mean squared error of probability predictions, measuring both calibration and sharpness of probabilistic classifiers.
AML Quality Assurance & Improvement 2
Which cross-validation strategy is most appropriate when your dataset has significant temporal ordering?