Model Evaluation and Validation Flashcards
7 cards from real MS-DS Master of Data science practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.
Read the first 7 Model Evaluation and Validation flashcards as text
Which of the following best describes the concept of 'data leakage' in model evaluation?
Answer: Information from outside the training window improperly influencing the model
Data leakage occurs when information that would not be available at prediction time is used during training, leading to overly optimistic evaluation metrics.
The DeLong method is used to:
Answer: Statistically compare two AUC-ROC values from correlated samples
The DeLong method provides a non-parametric test and confidence intervals for comparing two ROC AUC scores, accounting for correlation when models are evaluated on the same test set.
What is the key advantage of stratified k-fold cross-validation over standard k-fold?
Answer: It ensures each fold preserves the original class distribution
Stratified k-fold ensures each fold has approximately the same proportion of each class as the full dataset, which is crucial for imbalanced classification problems.
Mean Absolute Percentage Error (MAPE) is problematic when:
Answer: The target variable has negative values or values near zero
MAPE divides by the actual value, so zero or near-zero actuals cause division by zero or extremely large errors, and it is undefined for negative targets.
In the bias-variance decomposition of MSE, which term represents the model's sensitivity to fluctuations in the training data?
Answer: Variance
Variance quantifies how much the model's predictions change across different training sets drawn from the same distribution, reflecting sensitivity to training data fluctuations.
A Bland-Altman plot is used in model validation to:
Answer: Assess agreement between two continuous measurement methods
A Bland-Altman plot displays the difference between two measurements against their mean, revealing systematic biases and limits of agreement in regression or measurement models.
Temporal cross-validation (walk-forward validation) differs from standard CV because it:
Answer: Always trains on past data and tests on future data within each fold
Walk-forward validation respects temporal order by expanding the training window and always predicting forward in time, preventing future information from entering training.