AML Model Evaluation & Optimization Techniques 2 — Questions and Answers
Question 1: Which technique addresses class imbalance during model evaluation by computing metrics on a resampled dataset that reflects equal class distribution?
- Stratified k-fold cross-validation
- SMOTE oversampling followed by standard evaluation
- Bootstrap resampling with replacement
- Balanced accuracy metric on original data (Correct answer)
Correct answer: Balanced accuracy metric on original data
Balanced accuracy averages recall across classes, directly accounting for imbalance without resampling the dataset itself.
Question 2: In Bayesian hyperparameter optimization, what probabilistic model is most commonly used to approximate the objective function?
- Random forest surrogate
- Gaussian Process (Correct answer)
- Neural network meta-learner
- Support vector regressor
Correct answer: Gaussian Process
Gaussian Processes provide a posterior distribution over the objective function, enabling uncertainty-aware acquisition functions like Expected Improvement.
Question 3: What does a DET (Detection Error Tradeoff) curve plot that distinguishes it from an ROC curve?
- Precision vs. recall on a linear scale
- False Negative Rate vs. False Positive Rate on normal deviate axes (Correct answer)
- True Positive Rate vs. False Positive Rate on log scale
- Lift vs. cumulative population percentage
Correct answer: False Negative Rate vs. False Positive Rate on normal deviate axes
DET curves plot FNR vs. FPR using normal deviate (probit) scales, which linearize performance curves for easier comparison of classifiers.
Question 4: When using nested cross-validation for model selection, what does the outer loop estimate?
- The optimal hyperparameter values
- The generalization error of the model selection procedure (Correct answer)
- The training error on the full dataset
- The variance of individual predictions
Correct answer: The generalization error of the model selection procedure
The outer loop in nested cross-validation provides an unbiased estimate of the generalization error of the entire model selection pipeline, not just the selected model.
Question 5: Which optimizer adapts per-parameter learning rates using exponentially weighted moving averages of both gradients and their squares?
- RMSprop
- AdaGrad
- Adam (Correct answer)
- SGD with momentum
Correct answer: Adam
Adam combines first-moment (gradient) and second-moment (squared gradient) exponential moving averages with bias correction, making it more adaptive than RMSprop or AdaGrad alone.
Question 6: What is the primary purpose of the Expected Improvement (EI) acquisition function in Bayesian optimization?
- To directly minimize the surrogate model's prediction
- To balance exploration and exploitation by quantifying improvement probability over the current best (Correct answer)
- To sample uniformly from unexplored regions
- To ensemble predictions from multiple surrogates
Correct answer: To balance exploration and exploitation by quantifying improvement probability over the current best
EI computes the expected value of improvement over the current best observation, naturally trading off exploration (uncertainty) against exploitation (predicted value).
Question 7: A model achieves 95% accuracy but only 20% recall on the minority class in a fraud detection task. Which metric best captures this failure?
- F1-score on the majority class
- Matthews Correlation Coefficient (MCC) (Correct answer)
- Top-k accuracy
- Mean Average Precision
Correct answer: Matthews Correlation Coefficient (MCC)
MCC accounts for all four quadrants of the confusion matrix and is robust to class imbalance, making it a better single metric than accuracy or class-specific F1 in this scenario.
AML Model Evaluation & Optimization Techniques 2
Which technique addresses class imbalance during model evaluation by computing metrics on a resampled dataset that reflects equal class distribution?