Model Evaluation and Validation Flashcards

Which metric is most appropriate when false negatives are far more costly than false positives, such as in cancer screening?

Answer: Recall

Recall (sensitivity) measures how many actual positives are correctly identified, minimizing false negatives.

What does a Precision-Recall curve's Area Under the Curve (AUC-PR) near 0.5 indicate for a balanced dataset?

Answer: The model performs at chance level

For a balanced dataset, a random classifier achieves AUC-PR near 0.5, so scores near that value suggest poor discriminative ability.

In stratified k-fold cross-validation, what property is preserved in each fold?

Answer: The class distribution

Stratified k-fold ensures each fold reflects the overall class proportions, which is critical for imbalanced datasets.

What is the main purpose of a validation set as distinct from both training and test sets?

Answer: To tune hyperparameters without biasing test evaluation

The validation set is used for hyperparameter tuning so the test set remains unseen and provides an unbiased generalization estimate.

A model achieves 99% accuracy on a dataset where 99% of samples belong to one class. This is an example of:

Answer: The accuracy paradox

The accuracy paradox occurs when high accuracy is misleading because a naive baseline (predicting the majority class always) achieves the same score.

Which evaluation approach is most suitable for evaluating a time-series forecasting model?

Answer: Walk-forward (rolling) validation

Walk-forward validation respects temporal ordering by always training on past data and testing on future data, preventing data leakage.

The Matthews Correlation Coefficient (MCC) is preferred over accuracy for imbalanced binary classification because:

Answer: It accounts for all four confusion matrix values and handles class imbalance

MCC incorporates TP, TN, FP, and FN into a single balanced measure, making it robust even when classes are heavily imbalanced.