Machine Learning Model Evaluation 3 — Questions and Answers
Question 1: Which metric is best suited for evaluating regression models?
- F1 Score
- AUC-ROC
- Mean Squared Error (MSE) (Correct answer)
- Matthews Correlation Coefficient
Correct answer: Mean Squared Error (MSE)
MSE measures the average squared difference between predicted and actual continuous values, making it standard for regression evaluation.
Question 2: What is the key advantage of using Mean Absolute Error (MAE) over Mean Squared Error (MSE)?
- MAE is always smaller than MSE
- MAE is less sensitive to outliers (Correct answer)
- MAE penalizes large errors more heavily
- MAE is differentiable everywhere
Correct answer: MAE is less sensitive to outliers
MAE uses absolute differences rather than squared differences, making it more robust to outliers compared to MSE.
Question 3: What does the R² (coefficient of determination) value of 0 indicate?
- The model perfectly predicts outcomes
- The model performs as well as a constant mean predictor (Correct answer)
- The model has negative predictive power
- The residuals are normally distributed
Correct answer: The model performs as well as a constant mean predictor
R²=0 means the model explains none of the variance in the target variable, performing no better than simply predicting the mean.
Question 4: What is 'data leakage' in the context of model evaluation?
- Training data accidentally deleted
- Test set information influencing the training process (Correct answer)
- Model weights being exposed publicly
- Overfitting to the validation set
Correct answer: Test set information influencing the training process
Data leakage occurs when information from the test/future data leaks into training, causing overly optimistic evaluation metrics.
Question 5: Which technique helps evaluate model stability across multiple different random train/test splits?
- Single hold-out split
- Repeated cross-validation (Correct answer)
- Grid search
- Learning curve analysis
Correct answer: Repeated cross-validation
Repeated cross-validation runs k-fold CV multiple times with different random splits to assess how stable performance estimates are.
Question 6: When comparing two models using cross-validation scores, which statistical test is commonly used?
- Chi-squared test
- Paired t-test (Correct answer)
- ANOVA
- Z-test for proportions
Correct answer: Paired t-test
A paired t-test compares cross-validation scores from the same folds across two models, accounting for the paired nature of the data.
Question 7: What does a learning curve showing high training error AND high validation error indicate?
- Overfitting
- Underfitting (Correct answer)
- Good generalization
- High variance
Correct answer: Underfitting
When both training and validation errors are high, the model is underfitting — it lacks capacity to capture the data's underlying patterns.
Machine Learning Model Evaluation 3
Which metric is best suited for evaluating regression models?