Machine Learning Model Evaluation 4 — Questions and Answers
Question 1: What is the Matthews Correlation Coefficient (MCC) particularly useful for?
- Regression tasks with outliers
- Evaluating classifiers on imbalanced datasets (Correct answer)
- Measuring clustering quality
- Comparing models across different domains
Correct answer: Evaluating classifiers on imbalanced datasets
MCC accounts for all four cells of the confusion matrix and provides a balanced measure even when classes are severely imbalanced.
Question 2: In a precision-recall curve, what does a model with high area under the curve (AUC-PR) signify?
- The model performs well only on the majority class
- The model maintains high precision while achieving high recall (Correct answer)
- The model is overfitted to training data
- The decision threshold is set optimally
Correct answer: The model maintains high precision while achieving high recall
High AUC-PR means the model achieves good precision across a range of recall values, indicating strong performance on the positive class.
Question 3: What is 'calibration' in the context of probabilistic classifiers?
- Tuning hyperparameters to maximize accuracy
- Ensuring predicted probabilities match observed event frequencies (Correct answer)
- Normalizing feature values before training
- Adjusting the decision boundary post-training
Correct answer: Ensuring predicted probabilities match observed event frequencies
A calibrated model produces predicted probabilities that reflect the true likelihood of outcomes (e.g., 70% predictions correct ~70% of the time).
Question 4: Which metric would you use to evaluate a ranking model (e.g., search engine results)?
- RMSE
- Accuracy
- Normalized Discounted Cumulative Gain (NDCG) (Correct answer)
- Cohen's Kappa
Correct answer: Normalized Discounted Cumulative Gain (NDCG)
NDCG evaluates the quality of ranked results by considering both relevance and position, discounting lower-ranked items.
Question 5: What is 'stratified k-fold cross-validation' designed to handle?
- Time-series data with temporal ordering
- Datasets with imbalanced class distributions (Correct answer)
- Regression problems with continuous targets
- Multi-label classification scenarios
Correct answer: Datasets with imbalanced class distributions
Stratified k-fold ensures each fold maintains the same class proportion as the full dataset, critical for imbalanced classification.
Question 6: What does 'early stopping' evaluate in the context of neural network training?
- Training loss reaching zero
- When validation performance stops improving (Correct answer)
- When gradients become too small
- When the learning rate reaches a minimum
Correct answer: When validation performance stops improving
Early stopping monitors validation loss during training and halts when it stops improving, preventing overfitting to training data.
Question 7: What is 'Cohen's Kappa' used to measure?
- Correlation between continuous variables
- Agreement between classifier predictions and actual labels, adjusted for chance (Correct answer)
- Variance explained by a regression model
- Distance between cluster centroids
Correct answer: Agreement between classifier predictions and actual labels, adjusted for chance
Cohen's Kappa measures inter-rater agreement adjusted for the agreement expected by chance, providing a more reliable accuracy measure.
Machine Learning Model Evaluation 4
What is the Matthews Correlation Coefficient (MCC) particularly useful for?