MS-DS Master of Data science Model Evaluation and Validation 5

Question 1

Which of the following is a consequence of using the test set multiple times during model development?

Accepted Answer

Optimistic bias in generalization estimates

Answer

Repeatedly evaluating on the test set allows implicit overfitting to it, making the reported metrics overly optimistic and no longer representative of true generalization.

Question 2

Lift in model evaluation is calculated as:

Accepted Answer

(Precision of model) / (Precision of random model)

Answer

Lift measures how much better the model performs compared to a random baseline, calculated as the model's response rate in a decile divided by the overall response rate.

Question 3

What is the purpose of a reliability diagram (calibration plot)?

Accepted Answer

To visualize whether predicted probabilities match observed event frequencies

Answer

A reliability diagram bins predictions by predicted probability and plots mean predicted probability versus observed frequency, revealing over- or under-confidence.

Question 4

In a multiclass classification problem, macro-averaged F1 differs from weighted-averaged F1 in that macro-averaging:

Accepted Answer

Gives equal weight to each class regardless of class frequency

Answer

Macro-averaging computes F1 for each class independently and takes an unweighted mean, treating all classes equally regardless of how many samples they contain.

Question 5

What does a learning curve showing high training error and high validation error indicate?

Accepted Answer

Underfitting — model lacks capacity to capture the pattern

Answer

When both training and validation errors are high and converge, the model is underfitting because it cannot capture the underlying pattern in the data.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Model Evaluation and Validation 5