DSE Model Evaluation and Validation 2 — Questions and Answers
Question 1: Which metric is most appropriate when false negatives are far more costly than false positives, such as in cancer screening?
- Precision
- Recall (Correct answer)
- Specificity
- Accuracy
Correct answer: Recall
Recall (sensitivity) measures how many actual positives are correctly identified, minimizing false negatives.
Question 2: What does a Precision-Recall curve's Area Under the Curve (AUC-PR) near 0.5 indicate for a balanced dataset?
- The model performs well
- The model performs at chance level (Correct answer)
- The model always predicts negative
- The model is overfitting
Correct answer: The model performs at chance level
For a balanced dataset, a random classifier achieves AUC-PR near 0.5, so scores near that value suggest poor discriminative ability.
Question 3: In stratified k-fold cross-validation, what property is preserved in each fold?
- The order of samples
- The class distribution (Correct answer)
- The feature variance
- The model hyperparameters
Correct answer: The class distribution
Stratified k-fold ensures each fold reflects the overall class proportions, which is critical for imbalanced datasets.
Question 4: What is the main purpose of a validation set as distinct from both training and test sets?
- To provide more training data
- To tune hyperparameters without biasing test evaluation (Correct answer)
- To measure final model generalization
- To detect data leakage
Correct answer: To tune hyperparameters without biasing test evaluation
The validation set is used for hyperparameter tuning so the test set remains unseen and provides an unbiased generalization estimate.
Question 5: A model achieves 99% accuracy on a dataset where 99% of samples belong to one class. This is an example of:
- Excellent model performance
- The accuracy paradox (Correct answer)
- Overfitting to training data
- A well-calibrated model
Correct answer: The accuracy paradox
The accuracy paradox occurs when high accuracy is misleading because a naive baseline (predicting the majority class always) achieves the same score.
Question 6: Which evaluation approach is most suitable for evaluating a time-series forecasting model?
- Standard k-fold cross-validation
- Leave-one-out cross-validation
- Walk-forward (rolling) validation (Correct answer)
- Bootstrap resampling
Correct answer: Walk-forward (rolling) validation
Walk-forward validation respects temporal ordering by always training on past data and testing on future data, preventing data leakage.
Question 7: The Matthews Correlation Coefficient (MCC) is preferred over accuracy for imbalanced binary classification because:
- It is faster to compute
- It accounts for all four confusion matrix values and handles class imbalance (Correct answer)
- It only considers true positives and true negatives
- It is equivalent to the F1-score
Correct answer: It accounts for all four confusion matrix values and handles class imbalance
MCC incorporates TP, TN, FP, and FN into a single balanced measure, making it robust even when classes are heavily imbalanced.
DSE Model Evaluation and Validation 2
Which metric is most appropriate when false negatives are far more costly than false positives, such as in cancer screening?