DSE Model Evaluation and Validation 3

Question 1

What does the Brier Score measure in probabilistic classification models?

Accepted Answer

Mean squared error between predicted probabilities and actual outcomes

Answer

The Brier Score is the mean squared difference between predicted probabilities and binary outcomes, where lower scores indicate better-calibrated predictions.

Question 2

In a reliability diagram (calibration plot), a perfectly calibrated model's curve would:

Accepted Answer

Lie along the diagonal from (0,0) to (1,1)

Answer

Perfect calibration means that when a model predicts 70% probability, 70% of those cases are truly positive, forming a diagonal line.

Question 3

Which technique adjusts a classifier's predicted probabilities to be better calibrated without retraining it?

Accepted Answer

Platt scaling

Answer

Platt scaling fits a logistic regression on the model's raw outputs to transform them into calibrated probabilities.

Question 4

When using RMSE versus MAE to evaluate regression models, RMSE is preferred when:

Accepted Answer

Large errors are particularly undesirable

Answer

RMSE squares errors before averaging, so it penalizes large errors more heavily than MAE, making it preferable when large deviations are costly.

Question 5

What is the key difference between R² (coefficient of determination) and Adjusted R²?

Accepted Answer

Adjusted R² penalizes for the number of predictors added

Answer

Adjusted R² penalizes for adding irrelevant predictors, unlike R² which always increases or stays the same as features are added.

DSE - Data Science Practice Test

DSE - Data Science Practice Test

DSE Model Evaluation and Validation 3