FREE Data Science Model Evaluation and Validation Questions and Answers Flashcards

Which technique helps detect data leakage during model validation?

Answer: Comparing training performance with cross-validation performance

A suspiciously large gap where training performance is near-perfect while cross-validation performance is poor often signals data leakage.

What is the purpose of nested cross-validation?

Answer: To simultaneously perform hyperparameter tuning and unbiased performance estimation

Nested cross-validation uses an inner loop for hyperparameter selection and an outer loop for performance estimation, preventing optimistic bias from tuning on the evaluation set.

Which evaluation metric directly accounts for both the ranking quality and calibration of predicted probabilities?

Answer: Log Loss

Log loss penalizes both incorrect rankings and poorly calibrated probability estimates, making it sensitive to the confidence level of predictions.

In time series model validation, why is standard k-fold cross-validation inappropriate?

Answer: It violates temporal ordering by using future data to predict the past

Standard k-fold ignores temporal dependencies, potentially training on future observations to predict past ones, which inflates performance estimates unrealistically.

What does McNemar's test evaluate when comparing two classifiers?

Answer: Whether they disagree on predictions in a statistically significant way

McNemar's test analyzes the contingency table of disagreements between two classifiers to determine if their error patterns differ significantly.

What is the effect of using too few folds in k-fold cross-validation?

Answer: Higher bias in the performance estimate due to smaller training sets

Fewer folds mean each training set is smaller relative to the full dataset, leading to pessimistically biased performance estimates.