Master of Data science Model Evaluation and Validation 1 Flashcards
6 cards from real MS-DS Master of Data science practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.
Read the first 6 Master of Data science Model Evaluation and Validation 1 flashcards as text
Which of the following best describes the purpose of a calibration curve in model evaluation?
Answer: To compare predicted probabilities against actual outcome frequencies
A calibration curve (reliability diagram) plots the mean predicted probability against the actual fraction of positive outcomes in bins, revealing whether a model's confidence scores are trustworthy — a perfectly calibrated model falls on the diagonal.
You train a logistic regression model and observe AUC-ROC = 0.97 on the training set but AUC-ROC = 0.61 on the test set. Which action most directly addresses this problem?
Answer: Apply stronger regularization (increase λ)
A large gap between training and test AUC-ROC indicates overfitting (high variance). Increasing the regularization strength penalizes model complexity, reducing the gap between train and test performance.
What does the Brier Score measure in probabilistic model evaluation?
Answer: The mean squared error between predicted probabilities and actual binary outcomes
The Brier Score computes the mean squared difference between the predicted probability and the true binary label (0 or 1). Lower scores indicate better-calibrated probabilistic predictions, with 0 being perfect and 1 being worst.
In a multi-class classification problem with severe class imbalance, which averaging strategy for F1-score best accounts for class frequency when reporting overall model performance?
Answer: Weighted-average
Weighted-average F1-score calculates F1 for each class independently and then takes the average weighted by the number of true instances per class, making it sensitive to class imbalance and giving more influence to larger classes.
Which technique is specifically designed to estimate the generalization error of a model when the available dataset is very small (e.g., fewer than 50 samples)?
Answer: Leave-one-out cross-validation (LOOCV)
LOOCV uses every single sample as a test set exactly once, maximizing training data usage (n-1 samples per fold). This makes it ideal for very small datasets where a standard train/test split would leave too few samples for reliable training or evaluation.
A data scientist reports that their model achieves 95% accuracy on a binary classification task. The dataset contains 95% negative and 5% positive examples. What is the most critical concern about this result?
Answer: The accuracy metric is misleading because a trivial classifier predicting all negatives achieves the same score
When the majority class represents 95% of the data, a naïve classifier that always predicts the negative class will also achieve 95% accuracy without learning anything meaningful. Metrics such as precision, recall, F1-score, or AUC-ROC are needed to reveal true performance.