Model Evaluation and Validation Flashcards

A data scientist is developing a classification model to predict customer churn. The dataset is highly imbalanced, with only 5% of customers having churned. Which evaluation metric is most appropriate to assess the model's performance in identifying the minority class (churners)?

Answer: Recall (Sensitivity)

In imbalanced datasets, accuracy can be misleading. A model could achieve high accuracy by simply predicting the majority class for all instances. Recall, or Sensitivity, measures the model's ability to correctly identify all relevant instances of the positive class (churners in this case). It is calculated as TP / (TP + FN), making it crucial when the cost of false negatives (failing to identify a churner) is high.

Which of the following best describes the purpose of K-Fold Cross-Validation?

Answer: To obtain a more robust estimate of a model's performance on unseen data by using the entire dataset for both training and validation.

K-Fold Cross-Validation is a resampling procedure used to evaluate machine learning models. The dataset is partitioned into 'K' subsets (folds). The model is trained on K-1 folds and validated on the remaining fold. This process is repeated K times, with each fold serving as the validation set exactly once. The results are then averaged to provide a more reliable estimate of the model's generalization performance compared to a single train-test split.

A machine learning model exhibits high variance but low bias. What is the most likely characteristic of this model's performance?

Answer: The model is highly sensitive to small fluctuations in the training data, likely causing overfitting.

The bias-variance tradeoff is a central concept in machine learning. A model with high variance is highly flexible and captures a lot of the detail in the training data, including the noise. This sensitivity to the training data means its performance can fluctuate significantly with different training sets, a classic sign of overfitting. Low bias means the model's average prediction is close to the correct value, but the high variance leads to poor generalization on unseen data.

In the context of a binary classification model, what does the Area Under the ROC Curve (AUC) represent?

Answer: The probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

The AUC represents the measure of a model's ability to distinguish between classes. An AUC of 1.0 indicates a perfect classifier, while an AUC of 0.5 suggests the model performs no better than random guessing. It is interpreted as the probability that a classifier will assign a higher score to a randomly chosen positive example than to a randomly chosen negative example.

A team is building a spam filter. The primary goal is to minimize the number of legitimate emails that are incorrectly classified as spam. Which evaluation metric should the team prioritize for optimization?

Answer: Precision

Precision measures the proportion of predicted positives that were actually correct (TP / (TP + FP)). In this scenario, a false positive occurs when a legitimate email (ham) is incorrectly classified as spam. Minimizing these false positives is the main goal, which directly translates to maximizing precision.

What is a primary drawback of using accuracy as the sole evaluation metric for a classification model, especially in real-world applications?

Answer: It can be highly misleading on datasets with a significant class imbalance.

Accuracy calculates the ratio of correct predictions to the total number of predictions. In a dataset where one class dominates (e.g., 99% Class A, 1% Class B), a model can achieve 99% accuracy by simply always predicting the majority class (Class A). This gives a false sense of high performance while the model is completely failing to identify the minority class.