MS-DS Master of Data science Model Evaluation and Validation 4 — Questions and Answers
Question 1: Which of the following best describes the concept of 'data leakage' in model evaluation?
- Training data accidentally deleted during preprocessing
- Information from outside the training window improperly influencing the model (Correct answer)
- Random noise added to features during augmentation
- Features with low variance being dropped before evaluation
Correct answer: Information from outside the training window improperly influencing the model
Data leakage occurs when information that would not be available at prediction time is used during training, leading to overly optimistic evaluation metrics.
Question 2: The DeLong method is used to:
- Compute confidence intervals for regression RMSE
- Statistically compare two AUC-ROC values from correlated samples (Correct answer)
- Perform multiple hypothesis correction for model metrics
- Estimate the optimal classification threshold
Correct answer: Statistically compare two AUC-ROC values from correlated samples
The DeLong method provides a non-parametric test and confidence intervals for comparing two ROC AUC scores, accounting for correlation when models are evaluated on the same test set.
Question 3: What is the key advantage of stratified k-fold cross-validation over standard k-fold?
- It is computationally faster for large datasets
- It ensures each fold preserves the original class distribution (Correct answer)
- It eliminates the need for a separate test set
- It prevents overfitting by randomly shuffling the data
Correct answer: It ensures each fold preserves the original class distribution
Stratified k-fold ensures each fold has approximately the same proportion of each class as the full dataset, which is crucial for imbalanced classification problems.
Question 4: Mean Absolute Percentage Error (MAPE) is problematic when:
- The target variable has negative values or values near zero (Correct answer)
- The model has more than 10 features
- Training and test sets are from different distributions
- The target variable is log-normally distributed
Correct answer: The target variable has negative values or values near zero
MAPE divides by the actual value, so zero or near-zero actuals cause division by zero or extremely large errors, and it is undefined for negative targets.
Question 5: In the bias-variance decomposition of MSE, which term represents the model's sensitivity to fluctuations in the training data?
- Bias²
- Variance (Correct answer)
- Irreducible error
- Regularization term
Correct answer: Variance
Variance quantifies how much the model's predictions change across different training sets drawn from the same distribution, reflecting sensitivity to training data fluctuations.
Question 6: A Bland-Altman plot is used in model validation to:
- Visualize the ROC curve for binary classifiers
- Assess agreement between two continuous measurement methods (Correct answer)
- Plot learning curves as training size increases
- Display feature importance as model complexity grows
Correct answer: Assess agreement between two continuous measurement methods
A Bland-Altman plot displays the difference between two measurements against their mean, revealing systematic biases and limits of agreement in regression or measurement models.
Question 7: Temporal cross-validation (walk-forward validation) differs from standard CV because it:
- Uses more folds to reduce variance
- Always trains on past data and tests on future data within each fold (Correct answer)
- Removes seasonality before splitting data
- Applies bootstrapping instead of deterministic splits
Correct answer: Always trains on past data and tests on future data within each fold
Walk-forward validation respects temporal order by expanding the training window and always predicting forward in time, preventing future information from entering training.
MS-DS Master of Data science Model Evaluation and Validation 4
Which of the following best describes the concept of 'data leakage' in model evaluation?