Data Science Model Performance and Evaluation Questions and Answers 1

Question 1

A medical research team develops a model to screen for a rare but aggressive form of cancer. The consequences of failing to identify a person who has the cancer (a false negative) are far more severe than mistakenly flagging a healthy person for additional testing (a false positive). Which evaluation metric is most critical to maximize for this model?

Accepted Answer

Recall (Sensitivity)

Answer

Recall, also known as Sensitivity, measures the model's ability to correctly identify all actual positive cases (True Positives / (True Positives + False Negatives)). In this medical scenario, minimizing false negatives is the top priority to ensure patients with cancer receive timely treatment. Therefore, maximizing recall is the most critical objective.

Question 2

Which of the following statements best describes a key difference between Mean Squared Error (MSE) and Mean Absolute Error (MAE) as regression model evaluation metrics?

Accepted Answer

MSE penalizes larger errors more heavily than MAE.

Answer

MSE calculates the average of the squared differences between predicted and actual values. By squaring the error term, it gives a disproportionately larger weight to significant errors (outliers). MAE, which takes the average of the absolute differences, treats all errors with a weight proportional to their magnitude. Therefore, MSE penalizes large errors more severely.

Question 3

A data scientist has a very small dataset containing only 150 samples and needs to evaluate the performance of a predictive model. Which validation strategy would provide the most reliable estimate of the model's generalization ability?

Accepted Answer

K-Fold Cross-Validation.

Answer

With a small dataset, a single train-test split is highly dependent on which specific samples end up in the training and test sets, leading to a high-variance performance estimate. K-Fold Cross-Validation is the preferred method as it allows every data point to be used in both a training and a testing fold across different iterations, providing a more robust and stable estimate of model performance on unseen data.

Question 4

The Area Under the ROC Curve (AUC) is a popular metric for classification models. What does an AUC score of 0.85 signify?

Accepted Answer

There is an 85% probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

Answer

The AUC has a specific statistical interpretation. It represents the probability that the classifier will assign a higher score or probability to a randomly selected positive instance than to a randomly selected negative instance. It is a measure of the model's ability to discriminate between the positive and negative classes across all possible thresholds.

Question 5

While training a deep learning model, a data scientist observes that the training accuracy steadily increases to 98%, while the validation accuracy increases for a while but then plateaus and starts to decrease, ending at 82%. This divergence in performance curves is a classic sign of what phenomenon?

Accepted Answer

Overfitting

Answer

Overfitting occurs when a model learns the training data too well, including its noise, and fails to generalize to new, unseen data. The key indicator is a large and growing gap between the model's performance on the training set (high accuracy) and its performance on the validation set (lower, and often decreasing, accuracy).

Data Science Practice Test

Data Science Practice Test

Data Science Model Performance and Evaluation Questions and Answers 1