DSE Model Evaluation and Validation 3 ā Questions and Answers
Question 1: What does the Brier Score measure in probabilistic classification models?
- Rank ordering of predictions
- Mean squared error between predicted probabilities and actual outcomes (Correct answer)
- Area under the ROC curve
- Calibration of confidence intervals
Correct answer: Mean squared error between predicted probabilities and actual outcomes
The Brier Score is the mean squared difference between predicted probabilities and binary outcomes, where lower scores indicate better-calibrated predictions.
Question 2: In a reliability diagram (calibration plot), a perfectly calibrated model's curve would:
- Follow a horizontal line at 0.5
- Lie along the diagonal from (0,0) to (1,1) (Correct answer)
- Show a steep S-curve
- Cluster near the y-axis
Correct answer: Lie along the diagonal from (0,0) to (1,1)
Perfect calibration means that when a model predicts 70% probability, 70% of those cases are truly positive, forming a diagonal line.
Question 3: Which technique adjusts a classifier's predicted probabilities to be better calibrated without retraining it?
- Grid search
- Platt scaling (Correct answer)
- SMOTE
- Principal Component Analysis
Correct answer: Platt scaling
Platt scaling fits a logistic regression on the model's raw outputs to transform them into calibrated probabilities.
Question 4: When using RMSE versus MAE to evaluate regression models, RMSE is preferred when:
- Outliers should be treated equally to other errors
- Large errors are particularly undesirable (Correct answer)
- The target variable is categorical
- Predictions need to be interpretable in original units
Correct answer: Large errors are particularly undesirable
RMSE squares errors before averaging, so it penalizes large errors more heavily than MAE, making it preferable when large deviations are costly.
Question 5: What is the key difference between RĀ² (coefficient of determination) and Adjusted RĀ²?
- Adjusted RĀ² penalizes for the number of predictors added (Correct answer)
- RĀ² is only for linear models while Adjusted RĀ² works for any model
- Adjusted RĀ² is always higher than RĀ²
- RĀ² accounts for model complexity while Adjusted RĀ² does not
Correct answer: Adjusted RĀ² penalizes for the number of predictors added
Adjusted RĀ² penalizes for adding irrelevant predictors, unlike RĀ² which always increases or stays the same as features are added.
Question 6: In nested cross-validation, the inner loop is used for:
- Final model testing
- Hyperparameter selection (Correct answer)
- Feature extraction
- Data augmentation
Correct answer: Hyperparameter selection
The inner loop performs hyperparameter tuning, while the outer loop provides an unbiased estimate of the tuned model's generalization error.
Question 7: Which situation best describes when leave-one-out cross-validation (LOOCV) is most justified?
- Very large datasets with millions of samples
- Very small datasets where every sample matters (Correct answer)
- Highly imbalanced class distributions
- Models with many hyperparameters to tune
Correct answer: Very small datasets where every sample matters
LOOCV is computationally expensive but maximally uses available data, making it most valuable when datasets are too small for larger holdout folds.
DSE Model Evaluation and Validation 3
What does the Brier Score measure in probabilistic classification models?