Data Science with Python Model Evaluation and Validation Flashcards

A data scientist develops a model to detect a rare but critical disease. The cost of failing to identify a sick patient (a false negative) is extremely high. Which evaluation metric should be prioritized to minimize the number of missed cases?

Answer: Recall (Sensitivity)

Recall, also known as Sensitivity or True Positive Rate, measures the model's ability to identify all actual positive cases. It is calculated as TP / (TP + FN). In a medical scenario where missing a case (a False Negative) is highly consequential, maximizing recall is the primary objective.

What is the primary purpose of using k-fold cross-validation in machine learning?

Answer: To provide a more robust and stable estimate of a model's performance on unseen data.

K-fold cross-validation involves repeatedly training and testing a model on different subsets ('folds') of the data. By averaging the performance scores from each fold, it provides a more reliable and less biased estimate of how the model will generalize to new, independent data compared to a single train-test split.

A machine learning model performs exceptionally well on the training data with 99% accuracy but its accuracy drops to 70% on the validation data. This phenomenon is a clear indicator of what issue?

Answer: Overfitting

Overfitting occurs when a model learns the training data too well, including its noise and random fluctuations, to the point that it cannot generalize to new, unseen data. A large performance gap between the training set and the validation/test set is the classic symptom of overfitting.

In the context of a binary classification model, which of the following statements best describes the bias-variance tradeoff?

Answer: Increasing model complexity typically decreases bias but increases variance.

The bias-variance tradeoff is a fundamental concept where bias represents errors from overly simplistic assumptions (underfitting), and variance represents errors from being too sensitive to the training data (overfitting). As a model's complexity increases, its ability to fit the training data improves, thus decreasing bias. However, this increased complexity makes it more likely to model the noise in the training data, thus increasing its variance.

A data scientist is building a spam filter. They want to ensure that when an email is flagged as spam, it is very likely to actually be spam, minimizing the chance of legitimate emails ending up in the spam folder. Which metric should they focus on maximizing?

Answer: Precision

Precision measures the proportion of positive predictions that were actually correct. It is calculated as TP / (TP + FP). In this scenario, a False Positive (a legitimate email incorrectly marked as spam) is very costly. Maximizing precision directly addresses this by penalizing false positives.

What does the Area Under the ROC Curve (AUC) represent for a classification model?

Answer: The probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

The AUC score provides a single number summarizing the performance of a classifier across all possible classification thresholds. Its probabilistic interpretation is that it measures the likelihood that the model will assign a higher score (probability) to a randomly selected positive example than to a randomly selected negative example. An AUC of 1.0 represents a perfect model, while 0.5 represents a model with no discriminative ability.