DSE Model Evaluation and Validation 4

Question 1

What is 'data leakage' in the context of model validation?

Accepted Answer

Information from the test set influencing model training or evaluation

Answer

Data leakage occurs when information from outside the training boundary (e.g., test labels or future data) influences the model, causing overly optimistic results.

Question 2

A model trained on hospital A data is tested on hospital B data and performs much worse. This primarily illustrates:

Accepted Answer

Distribution shift / covariate shift

Answer

Distribution shift occurs when the statistical properties of the test environment differ from training, leading to performance degradation.

Question 3

The G-mean metric for imbalanced classification is calculated as:

Accepted Answer

√(Sensitivity × Specificity)

Answer

G-mean is the geometric mean of sensitivity and specificity, balancing performance across both the positive and negative classes.

Question 4

When evaluating a multi-class classifier, 'macro-averaged' F1-score differs from 'weighted-averaged' F1-score in that macro averaging:

Accepted Answer

Gives equal weight to all classes regardless of size

Answer

Macro-averaging computes the metric for each class independently and takes the unweighted mean, treating all classes equally regardless of support.

Question 5

Which of the following is a sign of high variance (overfitting) when examining learning curves?

Accepted Answer

Training error is low but validation error is much higher

Answer

A large gap between low training error and high validation error is the classic signature of overfitting (high variance).

DSE - Data Science Practice Test

DSE - Data Science Practice Test

DSE Model Evaluation and Validation 4