Data Science Model Performance and Evaluation Questions and Answers 1 — Questions and Answers
Question 1: A medical research team develops a model to screen for a rare but aggressive form of cancer. The consequences of failing to identify a person who has the cancer (a false negative) are far more severe than mistakenly flagging a healthy person for additional testing (a false positive). Which evaluation metric is most critical to maximize for this model?
- Accuracy
- Precision
- Recall (Sensitivity) (Correct answer)
- Specificity
Correct answer: Recall (Sensitivity)
Recall, also known as Sensitivity, measures the model's ability to correctly identify all actual positive cases (True Positives / (True Positives + False Negatives)). In this medical scenario, minimizing false negatives is the top priority to ensure patients with cancer receive timely treatment. Therefore, maximizing recall is the most critical objective.
Question 2: Which of the following statements best describes a key difference between Mean Squared Error (MSE) and Mean Absolute Error (MAE) as regression model evaluation metrics?
- MAE is more sensitive to outliers than MSE.
- MSE penalizes larger errors more heavily than MAE. (Correct answer)
- MAE is always a smaller value than MSE for the same set of predictions.
- MSE is easier to interpret because it is in the original units of the target variable.
Correct answer: MSE penalizes larger errors more heavily than MAE.
MSE calculates the average of the squared differences between predicted and actual values. By squaring the error term, it gives a disproportionately larger weight to significant errors (outliers). MAE, which takes the average of the absolute differences, treats all errors with a weight proportional to their magnitude. Therefore, MSE penalizes large errors more severely.
Question 3: A data scientist has a very small dataset containing only 150 samples and needs to evaluate the performance of a predictive model. Which validation strategy would provide the most reliable estimate of the model's generalization ability?
- A single 80/20 train-test split.
- K-Fold Cross-Validation. (Correct answer)
- Training on 100% of the data and testing on the same data.
- A single 50/50 train-test split.
Correct answer: K-Fold Cross-Validation.
With a small dataset, a single train-test split is highly dependent on which specific samples end up in the training and test sets, leading to a high-variance performance estimate. K-Fold Cross-Validation is the preferred method as it allows every data point to be used in both a training and a testing fold across different iterations, providing a more robust and stable estimate of model performance on unseen data.
Question 4: The Area Under the ROC Curve (AUC) is a popular metric for classification models. What does an AUC score of 0.85 signify?
- The model's accuracy is 85%.
- There is an 85% probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance. (Correct answer)
- The model correctly classifies 85% of the positive class instances.
- The F1-Score of the model is 0.85.
Correct answer: There is an 85% probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.
The AUC has a specific statistical interpretation. It represents the probability that the classifier will assign a higher score or probability to a randomly selected positive instance than to a randomly selected negative instance. It is a measure of the model's ability to discriminate between the positive and negative classes across all possible thresholds.
Question 5: While training a deep learning model, a data scientist observes that the training accuracy steadily increases to 98%, while the validation accuracy increases for a while but then plateaus and starts to decrease, ending at 82%. This divergence in performance curves is a classic sign of what phenomenon?
- Data leakage
- Underfitting
- Overfitting (Correct answer)
- High bias
Correct answer: Overfitting
Overfitting occurs when a model learns the training data too well, including its noise, and fails to generalize to new, unseen data. The key indicator is a large and growing gap between the model's performance on the training set (high accuracy) and its performance on the validation set (lower, and often decreasing, accuracy).
Question 6: When performing hyperparameter tuning for a complex model with many hyperparameters, what is a primary advantage of using Random Search over Grid Search?
- Random Search is guaranteed to find the optimal hyperparameter combination.
- Random Search is generally more computationally efficient because it does not waste evaluations on unimportant hyperparameters. (Correct answer)
- Grid Search cannot be used for models with more than three hyperparameters.
- Random Search explores every single possible combination of hyperparameter values.
Correct answer: Random Search is generally more computationally efficient because it does not waste evaluations on unimportant hyperparameters.
Grid Search exhaustively tries every specified combination, which can be computationally expensive. Random Search samples a fixed number of combinations from statistical distributions. If some hyperparameters are much more important than others, Random Search has a higher probability of finding a good setting for the important ones within the same computational budget, making it more efficient.
Data Science Model Performance and Evaluation Questions and Answers 1
A medical research team develops a model to screen for a rare but aggressive form of cancer.
The consequences of failing to identify a person who has the cancer (a false negative) are far more severe than mistakenly flagging a healthy person for additional testing (a false positive).
Which evaluation metric is most critical to maximize for this model?