Data Science FREE Data Science Model Evaluation and Validation Questions and Answers 3

Question 1

Which technique helps detect data leakage during model validation?

Accepted Answer

Comparing training performance with cross-validation performance

Answer

A suspiciously large gap where training performance is near-perfect while cross-validation performance is poor often signals data leakage.

Question 2

What is the purpose of nested cross-validation?

Accepted Answer

To simultaneously perform hyperparameter tuning and unbiased performance estimation

Answer

Nested cross-validation uses an inner loop for hyperparameter selection and an outer loop for performance estimation, preventing optimistic bias from tuning on the evaluation set.

Question 3

Which evaluation metric directly accounts for both the ranking quality and calibration of predicted probabilities?

Accepted Answer

Log Loss

Answer

Log loss penalizes both incorrect rankings and poorly calibrated probability estimates, making it sensitive to the confidence level of predictions.

Question 4

In time series model validation, why is standard k-fold cross-validation inappropriate?

Accepted Answer

It violates temporal ordering by using future data to predict the past

Answer

Standard k-fold ignores temporal dependencies, potentially training on future observations to predict past ones, which inflates performance estimates unrealistically.

Question 5

What does McNemar's test evaluate when comparing two classifiers?

Accepted Answer

Whether they disagree on predictions in a statistically significant way

Answer

McNemar's test analyzes the contingency table of disagreements between two classifiers to determine if their error patterns differ significantly.

Data Science Practice Test

Data Science Practice Test

Data Science FREE Data Science Model Evaluation and Validation Questions and Answers 3