Model Evaluation & Optimization Techniques Flashcards

Which technique addresses class imbalance during model evaluation by computing metrics on a resampled dataset that reflects equal class distribution?

Answer: Balanced accuracy metric on original data

Balanced accuracy averages recall across classes, directly accounting for imbalance without resampling the dataset itself.

In Bayesian hyperparameter optimization, what probabilistic model is most commonly used to approximate the objective function?

Answer: Gaussian Process

Gaussian Processes provide a posterior distribution over the objective function, enabling uncertainty-aware acquisition functions like Expected Improvement.

What does a DET (Detection Error Tradeoff) curve plot that distinguishes it from an ROC curve?

Answer: False Negative Rate vs. False Positive Rate on normal deviate axes

DET curves plot FNR vs. FPR using normal deviate (probit) scales, which linearize performance curves for easier comparison of classifiers.

When using nested cross-validation for model selection, what does the outer loop estimate?

Answer: The generalization error of the model selection procedure

The outer loop in nested cross-validation provides an unbiased estimate of the generalization error of the entire model selection pipeline, not just the selected model.

Which optimizer adapts per-parameter learning rates using exponentially weighted moving averages of both gradients and their squares?

Answer: Adam

Adam combines first-moment (gradient) and second-moment (squared gradient) exponential moving averages with bias correction, making it more adaptive than RMSprop or AdaGrad alone.

What is the primary purpose of the Expected Improvement (EI) acquisition function in Bayesian optimization?

Answer: To balance exploration and exploitation by quantifying improvement probability over the current best

EI computes the expected value of improvement over the current best observation, naturally trading off exploration (uncertainty) against exploitation (predicted value).

A model achieves 95% accuracy but only 20% recall on the minority class in a fraud detection task. Which metric best captures this failure?

Answer: Matthews Correlation Coefficient (MCC)

MCC accounts for all four quadrants of the confusion matrix and is robust to class imbalance, making it a better single metric than accuracy or class-specific F1 in this scenario.