MS-DS Master of Data science Model Evaluation and Validation 3

Question 1

When performing nested cross-validation, the outer loop is used for:

Accepted Answer

Unbiased generalization error estimation

Answer

Nested CV uses an inner loop for hyperparameter tuning and model selection, while the outer loop provides an unbiased estimate of the final model's generalization error.

Question 2

A regression model reports an R² of 0.85 on training data and 0.42 on test data. What is the most likely diagnosis?

Accepted Answer

Overfitting

Answer

A large gap between training R² and test R² indicates the model memorized training data patterns that do not generalize, a hallmark of overfitting.

Question 3

Which of the following is the correct interpretation of a 95% confidence interval for model accuracy?

Accepted Answer

If the experiment were repeated many times, 95% of such intervals would contain the true accuracy

Answer

A confidence interval is a frequentist concept meaning that the procedure generates intervals containing the true parameter 95% of the time across repeated experiments.

Question 4

What does the McNemar test evaluate in model comparison?

Accepted Answer

Whether the disagreements between two classifiers are statistically symmetric

Answer

McNemar's test uses a 2x2 contingency table of cases where classifiers disagree to test if one model significantly outperforms the other.

Question 5

In time-series model validation, why is standard k-fold cross-validation inappropriate?

Accepted Answer

It can cause temporal leakage by using future data to predict the past

Answer

Random splitting in k-fold CV allows future observations to appear in the training set, creating data leakage that violates the temporal structure of time-series data.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Model Evaluation and Validation 3