MS-DS Master of Data science Model Evaluation and Validation 3 — Questions and Answers
Question 1: When performing nested cross-validation, the outer loop is used for:
- Hyperparameter tuning
- Model selection
- Unbiased generalization error estimation (Correct answer)
- Feature selection only
Correct answer: Unbiased generalization error estimation
Nested CV uses an inner loop for hyperparameter tuning and model selection, while the outer loop provides an unbiased estimate of the final model's generalization error.
Question 2: A regression model reports an R² of 0.85 on training data and 0.42 on test data. What is the most likely diagnosis?
- Underfitting
- Overfitting (Correct answer)
- Data leakage in the test set
- Concept drift
Correct answer: Overfitting
A large gap between training R² and test R² indicates the model memorized training data patterns that do not generalize, a hallmark of overfitting.
Question 3: Which of the following is the correct interpretation of a 95% confidence interval for model accuracy?
- There is a 95% probability the true accuracy lies in this interval
- If the experiment were repeated many times, 95% of such intervals would contain the true accuracy (Correct answer)
- The model will be accurate 95% of the time on new data
- 95% of test samples were correctly classified
Correct answer: If the experiment were repeated many times, 95% of such intervals would contain the true accuracy
A confidence interval is a frequentist concept meaning that the procedure generates intervals containing the true parameter 95% of the time across repeated experiments.
Question 4: What does the McNemar test evaluate in model comparison?
- Whether two models have equal mean squared error
- Whether the disagreements between two classifiers are statistically symmetric (Correct answer)
- Whether a model's precision and recall are significantly different
- Whether the training and test distributions are the same
Correct answer: Whether the disagreements between two classifiers are statistically symmetric
McNemar's test uses a 2x2 contingency table of cases where classifiers disagree to test if one model significantly outperforms the other.
Question 5: In time-series model validation, why is standard k-fold cross-validation inappropriate?
- It requires too much computation for sequential data
- It can cause temporal leakage by using future data to predict the past (Correct answer)
- It assumes features are normally distributed
- It cannot handle regression problems
Correct answer: It can cause temporal leakage by using future data to predict the past
Random splitting in k-fold CV allows future observations to appear in the training set, creating data leakage that violates the temporal structure of time-series data.
Question 6: A confusion matrix shows TP=80, FP=20, FN=10, TN=90. What is the F1 score?
- 0.80
- 0.84 (Correct answer)
- 0.89
- 0.76
Correct answer: 0.84
Precision = 80/100 = 0.80, Recall = 80/90 ≈ 0.889; F1 = 2*(0.80*0.889)/(0.80+0.889) ≈ 0.842.
Question 7: What is the primary purpose of using a holdout validation set separate from both training and test sets?
- To increase the total amount of training data
- To tune hyperparameters without contaminating the final test evaluation (Correct answer)
- To detect outliers before model training
- To perform feature engineering independently
Correct answer: To tune hyperparameters without contaminating the final test evaluation
A separate validation set allows hyperparameter tuning while preserving the test set as a truly unseen benchmark for final model evaluation.
MS-DS Master of Data science Model Evaluation and Validation 3
When performing nested cross-validation, the outer loop is used for: