MS-DS Master of Data science Model Evaluation and Validation Questions and Answers — Questions and Answers
Question 1: A machine learning model is developed to detect a rare but critical disease. For this application, failing to identify a person who has the disease (a false negative) is a far more severe error than incorrectly flagging a healthy person (a false positive). Which evaluation metric should be prioritized for optimization?
- Accuracy
- Recall (Sensitivity) (Correct answer)
- Precision
- F1-Score
Correct answer: Recall (Sensitivity)
Recall, also known as Sensitivity or True Positive Rate, measures the model's ability to find all actual positive cases (TP / (TP + FN)). In a critical medical diagnosis scenario, maximizing recall is crucial to minimize the number of dangerous false negatives.
Question 2: What is the primary advantage of using k-fold cross-validation over a single train-test split for evaluating a model's performance?
- It provides a more robust and reliable estimate of the model's generalization performance by testing on multiple, different subsets of the data. (Correct answer)
- It significantly reduces the computational time required to train the model since each 'fold' is smaller than the full training set.
- It completely eliminates the possibility of overfitting during the training process.
- It automatically selects the best hyperparameters for the model without needing a separate validation set.
Correct answer: It provides a more robust and reliable estimate of the model's generalization performance by testing on multiple, different subsets of the data.
K-fold cross-validation provides a more stable estimate of a model's performance on unseen data. By training and testing the model on 'k' different subsets of the data and averaging the results, it reduces the variance associated with a single, potentially lucky or unlucky, train-test split.
Question 3: You are evaluating a regression model that predicts apartment rental prices in dollars. You need a metric that expresses the typical prediction error in dollars and penalizes larger errors more heavily. Which of the following metrics is the most suitable?
- Mean Absolute Error (MAE)
- R-squared (R²)
- Root Mean Squared Error (RMSE) (Correct answer)
- Mean Squared Error (MSE)
Correct answer: Root Mean Squared Error (RMSE)
RMSE is the best choice because it penalizes larger errors more significantly due to the squaring of errors, and the final square root operation converts the error metric back into the original units of the target variable (dollars), making it more interpretable than MSE.
Question 4: A data scientist trains a decision tree model. They observe that the model achieves 99% accuracy on the training data but only 70% accuracy on the test data. The performance on the training data is excellent, but the performance on the test data is significantly worse. This situation is a classic example of what?
- Data leakage
- High bias (Underfitting)
- A well-generalized model
- High variance (Overfitting) (Correct answer)
Correct answer: High variance (Overfitting)
High variance, or overfitting, occurs when a model learns the training data too well, including its noise and random fluctuations, rather than the underlying general patterns. This results in excellent performance on the training data but poor performance on new, unseen data (the test set).
Question 5: In the context of binary classification, which statement best describes what the Area Under the ROC Curve (AUC) represents?
- The model's overall accuracy at a default classification threshold of 0.5.
- The trade-off between the model's precision and recall.
- The total number of correct predictions made by the model.
- The probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance. (Correct answer)
Correct answer: The probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.
The AUC score is a measure of a classifier's ability to distinguish between classes. It is interpreted as the probability that the model assigns a higher score to a randomly chosen positive example than to a randomly chosen negative example. An AUC of 0.5 represents a model with no discrimination ability (random guessing), while an AUC of 1.0 represents a perfect classifier.
Question 6: When building a machine learning model, a data scientist splits the data into three sets: training, validation, and test. What is the specific and primary role of the validation set?
- To tune model hyperparameters and select the best performing model architecture before final evaluation. (Correct answer)
- To provide a final, unbiased assessment of the selected model's performance on completely unseen data.
- To be used for the primary training of the model's parameters (e.g., weights or coefficients).
- To check for and remove outliers and anomalies from the original dataset.
Correct answer: To tune model hyperparameters and select the best performing model architecture before final evaluation.
The validation set is used to fine-tune the model's hyperparameters (e.g., learning rate, number of trees) and to compare different models. The model is trained on the training set, evaluated and tuned on the validation set, and then the final, chosen model is evaluated once on the test set to get an unbiased estimate of its performance on new data.
