MS-DS Master of Data science Model Evaluation and Validation 4

Question 1

Which of the following best describes the concept of 'data leakage' in model evaluation?

Accepted Answer

Information from outside the training window improperly influencing the model

Answer

Data leakage occurs when information that would not be available at prediction time is used during training, leading to overly optimistic evaluation metrics.

Question 2

The DeLong method is used to:

Accepted Answer

Statistically compare two AUC-ROC values from correlated samples

Answer

The DeLong method provides a non-parametric test and confidence intervals for comparing two ROC AUC scores, accounting for correlation when models are evaluated on the same test set.

Question 3

What is the key advantage of stratified k-fold cross-validation over standard k-fold?

Accepted Answer

It ensures each fold preserves the original class distribution

Answer

Stratified k-fold ensures each fold has approximately the same proportion of each class as the full dataset, which is crucial for imbalanced classification problems.

Question 4

Mean Absolute Percentage Error (MAPE) is problematic when:

Accepted Answer

The target variable has negative values or values near zero

Answer

MAPE divides by the actual value, so zero or near-zero actuals cause division by zero or extremely large errors, and it is undefined for negative targets.

Question 5

In the bias-variance decomposition of MSE, which term represents the model's sensitivity to fluctuations in the training data?

Accepted Answer

Variance

Answer

Variance quantifies how much the model's predictions change across different training sets drawn from the same distribution, reflecting sensitivity to training data fluctuations.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Model Evaluation and Validation 4