MS-DS Master of Data science Model Evaluation and Validation 2

Question 1

Which evaluation metric is most appropriate when the cost of a false negative is much higher than the cost of a false positive?

Accepted Answer

Recall

Answer

Recall (sensitivity) measures the proportion of actual positives correctly identified, making it critical when missing a positive case (false negative) is very costly.

Question 2

In k-fold cross-validation, what happens as k increases toward n (leave-one-out CV)?

Accepted Answer

Bias decreases and variance increases

Answer

As k approaches n, each fold uses nearly all data for training (low bias), but the estimates become highly variable across folds (high variance).

Question 3

The Brier score is used to evaluate which type of model output?

Accepted Answer

Probability estimates from a probabilistic classifier

Answer

The Brier score measures the mean squared difference between predicted probabilities and actual binary outcomes, assessing probabilistic calibration.

Question 4

A model achieves 99% accuracy on a dataset where 99% of examples are negative. This is an example of:

Accepted Answer

The accuracy paradox

Answer

The accuracy paradox occurs when high accuracy is misleading because a naive model predicting the majority class always scores well on imbalanced datasets.

Question 5

What does the area under the Precision-Recall curve (AUC-PR) measure?

Accepted Answer

The overall quality of ranked probability predictions for imbalanced datasets

Answer

AUC-PR summarizes the precision-recall tradeoff across all thresholds and is preferred over AUC-ROC when classes are highly imbalanced.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Model Evaluation and Validation 2