DSE - Data Science Model Evaluation and Validation Questions and Answers

Question 1

A data scientist is developing a classification model to predict customer churn. The dataset is highly imbalanced, with only 5% of customers having churned. Which evaluation metric is most appropriate to assess the model's performance in identifying the minority class (churners)?

Accepted Answer

Recall (Sensitivity)

Answer

In imbalanced datasets, accuracy can be misleading. A model could achieve high accuracy by simply predicting the majority class for all instances. Recall, or Sensitivity, measures the model's ability to correctly identify all relevant instances of the positive class (churners in this case). It is calculated as TP / (TP + FN), making it crucial when the cost of false negatives (failing to identify a churner) is high.

Question 2

Which of the following best describes the purpose of K-Fold Cross-Validation?

Accepted Answer

To obtain a more robust estimate of a model's performance on unseen data by using the entire dataset for both training and validation.

Answer

K-Fold Cross-Validation is a resampling procedure used to evaluate machine learning models. The dataset is partitioned into 'K' subsets (folds). The model is trained on K-1 folds and validated on the remaining fold. This process is repeated K times, with each fold serving as the validation set exactly once. The results are then averaged to provide a more reliable estimate of the model's generalization performance compared to a single train-test split.

Question 3

A machine learning model exhibits high variance but low bias. What is the most likely characteristic of this model's performance?

Accepted Answer

The model is highly sensitive to small fluctuations in the training data, likely causing overfitting.

Answer

The bias-variance tradeoff is a central concept in machine learning. A model with high variance is highly flexible and captures a lot of the detail in the training data, including the noise. This sensitivity to the training data means its performance can fluctuate significantly with different training sets, a classic sign of overfitting. Low bias means the model's average prediction is close to the correct value, but the high variance leads to poor generalization on unseen data.

Question 4

In the context of a binary classification model, what does the Area Under the ROC Curve (AUC) represent?

Accepted Answer

The probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

Answer

The AUC represents the measure of a model's ability to distinguish between classes. An AUC of 1.0 indicates a perfect classifier, while an AUC of 0.5 suggests the model performs no better than random guessing. It is interpreted as the probability that a classifier will assign a higher score to a randomly chosen positive example than to a randomly chosen negative example.

Question 5

A team is building a spam filter. The primary goal is to minimize the number of legitimate emails that are incorrectly classified as spam. Which evaluation metric should the team prioritize for optimization?

Accepted Answer

Precision

Answer

Precision measures the proportion of predicted positives that were actually correct (TP / (TP + FP)). In this scenario, a false positive occurs when a legitimate email (ham) is incorrectly classified as spam. Minimizing these false positives is the main goal, which directly translates to maximizing precision.

DSE - Data Science Practice Test

DSE - Data Science Practice Test

DSE - Data Science Model Evaluation and Validation Questions and Answers