MS-DS Master of Data science MS-DS Master of Data science Model Evaluation and Validation Questions and Answers 2 — Questions and Answers
Question 1: In a stratified k-fold cross-validation, what is preserved in each fold compared to standard k-fold?
- The proportion of each class label (Correct answer)
- The ordering of observations
- The number of features selected
- The random seed used for splitting
Correct answer: The proportion of each class label
Stratified k-fold ensures each fold maintains the same class distribution as the original dataset, which is critical for imbalanced data.
Question 2: Which metric is most appropriate for evaluating a model when the cost of false negatives is significantly higher than false positives?
- Recall (Correct answer)
- Precision
- Accuracy
- Specificity
Correct answer: Recall
Recall measures the proportion of actual positives correctly identified, making it the priority when missing positive cases is costly.
Question 3: What does a high variance and low bias in a model's predictions typically indicate?
- The model is overfitting the training data (Correct answer)
- The model is underfitting the training data
- The model has reached optimal complexity
- The model needs more features
Correct answer: The model is overfitting the training data
High variance with low bias means the model fits training data very closely but fails to generalize, which is the hallmark of overfitting.
Question 4: When performing nested cross-validation, what is the purpose of the inner loop?
- Hyperparameter tuning (Correct answer)
- Final model evaluation
- Feature extraction
- Data augmentation
Correct answer: Hyperparameter tuning
The inner loop of nested cross-validation is used to select the best hyperparameters, while the outer loop provides an unbiased performance estimate.
Question 5: A calibration curve (reliability diagram) plots predicted probabilities against observed frequencies. What does a perfectly calibrated model's curve look like?
- A diagonal line from (0,0) to (1,1) (Correct answer)
- A horizontal line at y = 0.5
- An S-shaped sigmoid curve
- A vertical line at x = 0.5
Correct answer: A diagonal line from (0,0) to (1,1)
A perfectly calibrated model has predicted probabilities that exactly match observed frequencies, producing a diagonal line.
Question 6: Which resampling method provides the lowest variance estimate of model performance but can have pessimistic bias for small datasets?
- Leave-one-out cross-validation (Correct answer)
- Bootstrap with replacement
- Holdout validation with 50-50 split
- Repeated random subsampling
Correct answer: Leave-one-out cross-validation
Leave-one-out cross-validation uses nearly all data for training in each iteration, yielding low bias but can produce high variance for small samples and pessimistic estimates due to high correlation between folds.
MS-DS Master of Data science MS-DS Master of Data science Model Evaluation and Validation Questions and Answers 2
In a stratified k-fold cross-validation, what is preserved in each fold compared to standard k-fold?