DSE Model Evaluation and Validation 5 — Questions and Answers
Question 1: Bootstrap aggregating (bagging) reduces which component of the bias-variance tradeoff?
- Bias
- Variance (Correct answer)
- Irreducible error
- Both bias and variance equally
Correct answer: Variance
Bagging averages predictions from many models trained on bootstrapped samples, which reduces variance without substantially changing bias.
Question 2: Which scenario correctly illustrates optimistic bias in model evaluation?
- Using a separate held-out test set for final evaluation
- Reporting the best score from many random train/test splits without correction (Correct answer)
- Applying stratified k-fold cross-validation
- Using nested cross-validation for hyperparameter tuning
Correct answer: Reporting the best score from many random train/test splits without correction
Repeatedly splitting data and cherry-picking the best result inflates the apparent performance because lucky splits are over-represented.
Question 3: The Expected Calibration Error (ECE) is computed by:
- Averaging absolute differences between model confidence and accuracy across probability bins (Correct answer)
- Computing the area under the ROC curve
- Measuring the variance of predicted probabilities
- Summing squared residuals across all predictions
Correct answer: Averaging absolute differences between model confidence and accuracy across probability bins
ECE bins predictions by confidence and computes a weighted average of the absolute gap between mean confidence and observed accuracy in each bin.
Question 4: When should Mean Absolute Percentage Error (MAPE) be avoided as an evaluation metric?
- When the target variable has large values
- When true values are near or equal to zero (Correct answer)
- When comparing across different scales
- When the dataset contains outliers
Correct answer: When true values are near or equal to zero
MAPE divides by the true value, so near-zero targets cause division by near-zero and produce extremely large or undefined errors.
Question 5: Shapley values in model explanation primarily help with evaluation by:
- Reducing overfitting through regularization
- Attributing each feature's contribution to individual predictions fairly (Correct answer)
- Selecting the optimal number of cross-validation folds
- Measuring overall model calibration
Correct answer: Attributing each feature's contribution to individual predictions fairly
Shapley values from game theory fairly distribute prediction credit among features, enabling instance-level and global feature importance evaluation.
Question 6: What does a high AUC-ROC score but low AUC-PR score on the same model most likely indicate?
- The model is well-suited for the task
- The dataset is highly imbalanced and the model struggles with the minority class (Correct answer)
- The model is overfitting
- The features are poorly engineered
Correct answer: The dataset is highly imbalanced and the model struggles with the minority class
AUC-ROC can be misleadingly high on imbalanced data because TN dominates, while AUC-PR focuses on the minority class performance and reveals weaknesses.
Question 7: In shadow deployment for model validation in production, the new model:
- Replaces the old model immediately after offline validation
- Runs in parallel receiving real traffic but its outputs are not served to users (Correct answer)
- Is tested on a small random sample of live users
- Uses synthetic data to simulate production conditions
Correct answer: Runs in parallel receiving real traffic but its outputs are not served to users
Shadow deployment lets the new model process real traffic silently alongside the production model, enabling risk-free comparison of live performance.
DSE Model Evaluation and Validation 5
Bootstrap aggregating (bagging) reduces which component of the bias-variance tradeoff?