Machine Learning Model Evaluation 5 — Questions and Answers
Question 1: What is the 'bias-variance trade-off' in model evaluation?
- The balance between training speed and model accuracy
- The tension between underfitting (high bias) and overfitting (high variance) (Correct answer)
- The trade-off between precision and recall
- The balance between model complexity and interpretability
Correct answer: The tension between underfitting (high bias) and overfitting (high variance)
High bias leads to underfitting while high variance leads to overfitting; the trade-off is finding the model complexity that minimizes both.
Question 2: Which cross-validation approach is most appropriate for time-series data?
- Standard k-fold
- Stratified k-fold
- Walk-forward (time-series) cross-validation (Correct answer)
- LOOCV
Correct answer: Walk-forward (time-series) cross-validation
Walk-forward validation respects temporal order by always training on past data and testing on future data, preventing look-ahead bias.
Question 3: What does 'BLEU score' measure in NLP model evaluation?
- Sentiment accuracy of text classifiers
- Similarity between machine-generated and reference translations (Correct answer)
- Perplexity of language models
- Named entity recognition precision
Correct answer: Similarity between machine-generated and reference translations
BLEU (Bilingual Evaluation Understudy) compares n-gram overlap between machine-generated text and reference translations.
Question 4: What is the 'Brier score' used to evaluate?
- Classification accuracy on binary problems
- Ranking quality of recommendation systems
- Probabilistic prediction quality (calibration and resolution) (Correct answer)
- Clustering cohesion
Correct answer: Probabilistic prediction quality (calibration and resolution)
The Brier score measures the mean squared difference between predicted probabilities and actual binary outcomes, assessing probabilistic forecasts.
Question 5: When should you prefer macro-averaged F1 over micro-averaged F1 in multi-class classification?
- When all classes are balanced
- When you want each class to contribute equally regardless of size (Correct answer)
- When the largest class is most important
- When computational efficiency is a priority
Correct answer: When you want each class to contribute equally regardless of size
Macro-average computes F1 per class then averages them, giving equal weight to all classes regardless of their sample size.
Question 6: What does 'perplexity' measure in language model evaluation?
- The number of parameters in the model
- How well the model predicts a sample (lower = better) (Correct answer)
- The model's inference latency on a benchmark
- The vocabulary size the model was trained on
Correct answer: How well the model predicts a sample (lower = better)
Perplexity measures how surprised a language model is by test data; lower perplexity indicates the model assigns higher probability to the test sequences.
Question 7: What is 'bootstrap aggregation' (bagging) primarily used to reduce in model evaluation?
- Bias in training
- Variance in predictions (Correct answer)
- Training time
- Feature dimensionality
Correct answer: Variance in predictions
Bagging trains multiple models on bootstrap samples and averages their predictions, reducing variance without significantly increasing bias.
Machine Learning Model Evaluation 5
What is the 'bias-variance trade-off' in model evaluation?