MS-DS Master of Data science Model Evaluation and Validation 5

Question 1

Which of the following is a consequence of using the test set multiple times during model development?

Accepted Answer

Optimistic bias in generalization estimates

Answer

Increased irreducible error

Answer

Higher training variance

Answer

Slower model convergence

Question 2

Lift in model evaluation is calculated as:

Accepted Answer

(Precision of model) / (Precision of random model)

Answer

AUC-ROC minus 0.5

Answer

True positive rate minus false positive rate

Answer

F1 score divided by accuracy

Question 3

What is the purpose of a reliability diagram (calibration plot)?

Accepted Answer

To visualize whether predicted probabilities match observed event frequencies

Answer

To show how training loss decreases over epochs

Answer

To compare two models' ROC curves on the same axes

Answer

To display the distribution of residuals across predicted values

Question 4

In a multiclass classification problem, macro-averaged F1 differs from weighted-averaged F1 in that macro-averaging:

Accepted Answer

Gives equal weight to each class regardless of class frequency

Answer

Weights each class by its support (number of samples)

Answer

Only considers the top two most frequent classes

Answer

Is equivalent to accuracy when classes are balanced

Question 5

What does a learning curve showing high training error and high validation error indicate?

Accepted Answer

Underfitting — model lacks capacity to capture the pattern

Answer

Overfitting — model is too complex

Answer

Data leakage in the validation set

Answer

Optimal bias-variance tradeoff achieved

Question 6

Platt scaling is a technique used to:

Accepted Answer

Convert a classifier's raw scores into calibrated probability estimates

Answer

Reduce model overfitting by shrinking coefficients

Answer

Scale features to unit variance before evaluation

Answer

Adjust predictions for class imbalance via threshold shifting

Question 7

When comparing multiple models across several datasets, the Friedman test followed by a Nemenyi post-hoc test is preferred over pairwise t-tests because:

Accepted Answer

It controls the family-wise error rate inflated by multiple comparisons

Answer

It requires fewer hyperparameter assumptions

Answer

It can handle continuous and categorical metrics simultaneously

Answer

It is faster to compute for large benchmark suites