MS-DS Master of Data science Master of Data science Model Evaluation and Validation 1

Question 1

Which of the following best describes the purpose of a calibration curve in model evaluation?

Accepted Answer

To compare predicted probabilities against actual outcome frequencies

Answer

A calibration curve (reliability diagram) plots the mean predicted probability against the actual fraction of positive outcomes in bins, revealing whether a model's confidence scores are trustworthy — a perfectly calibrated model falls on the diagonal.

Question 2

You train a logistic regression model and observe AUC-ROC = 0.97 on the training set but AUC-ROC = 0.61 on the test set. Which action most directly addresses this problem?

Accepted Answer

Apply stronger regularization (increase λ)

Answer

A large gap between training and test AUC-ROC indicates overfitting (high variance). Increasing the regularization strength penalizes model complexity, reducing the gap between train and test performance.

Question 3

What does the Brier Score measure in probabilistic model evaluation?

Accepted Answer

The mean squared error between predicted probabilities and actual binary outcomes

Answer

The Brier Score computes the mean squared difference between the predicted probability and the true binary label (0 or 1). Lower scores indicate better-calibrated probabilistic predictions, with 0 being perfect and 1 being worst.

Question 4

In a multi-class classification problem with severe class imbalance, which averaging strategy for F1-score best accounts for class frequency when reporting overall model performance?

Accepted Answer

Weighted-average

Answer

Weighted-average F1-score calculates F1 for each class independently and then takes the average weighted by the number of true instances per class, making it sensitive to class imbalance and giving more influence to larger classes.

Question 5

Which technique is specifically designed to estimate the generalization error of a model when the available dataset is very small (e.g., fewer than 50 samples)?

Accepted Answer

Leave-one-out cross-validation (LOOCV)

Answer

LOOCV uses every single sample as a test set exactly once, maximizing training data usage (n-1 samples per fold). This makes it ideal for very small datasets where a standard train/test split would leave too few samples for reliable training or evaluation.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Master of Data science Model Evaluation and Validation 1