DSE Model Evaluation and Validation 5

Question 1

Bootstrap aggregating (bagging) reduces which component of the bias-variance tradeoff?

Accepted Answer

Variance

Answer

Bagging averages predictions from many models trained on bootstrapped samples, which reduces variance without substantially changing bias.

Question 2

Which scenario correctly illustrates optimistic bias in model evaluation?

Accepted Answer

Reporting the best score from many random train/test splits without correction

Answer

Repeatedly splitting data and cherry-picking the best result inflates the apparent performance because lucky splits are over-represented.

Question 3

The Expected Calibration Error (ECE) is computed by:

Accepted Answer

Averaging absolute differences between model confidence and accuracy across probability bins

Answer

ECE bins predictions by confidence and computes a weighted average of the absolute gap between mean confidence and observed accuracy in each bin.

Question 4

When should Mean Absolute Percentage Error (MAPE) be avoided as an evaluation metric?

Accepted Answer

When true values are near or equal to zero

Answer

MAPE divides by the true value, so near-zero targets cause division by near-zero and produce extremely large or undefined errors.

Question 5

Shapley values in model explanation primarily help with evaluation by:

Accepted Answer

Attributing each feature's contribution to individual predictions fairly

Answer

Shapley values from game theory fairly distribute prediction credit among features, enabling instance-level and global feature importance evaluation.

DSE - Data Science Practice Test