Model Evaluation and Validation Flashcards
6 cards from real MS-DS Master of Data science practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.
Read the first 6 Model Evaluation and Validation flashcards as text
A machine learning model is developed to detect a rare but critical disease. For this application, failing to identify a person who has the disease (a false negative) is a far more severe error than incorrectly flagging a healthy person (a false positive). Which evaluation metric should be prioritized for optimization?
Answer: Recall (Sensitivity)
Recall, also known as Sensitivity or True Positive Rate, measures the model's ability to find all actual positive cases (TP / (TP + FN)). In a critical medical diagnosis scenario, maximizing recall is crucial to minimize the number of dangerous false negatives.
What is the primary advantage of using k-fold cross-validation over a single train-test split for evaluating a model's performance?
Answer: It provides a more robust and reliable estimate of the model's generalization performance by testing on multiple, different subsets of the data.
K-fold cross-validation provides a more stable estimate of a model's performance on unseen data. By training and testing the model on 'k' different subsets of the data and averaging the results, it reduces the variance associated with a single, potentially lucky or unlucky, train-test split.
You are evaluating a regression model that predicts apartment rental prices in dollars. You need a metric that expresses the typical prediction error in dollars and penalizes larger errors more heavily. Which of the following metrics is the most suitable?
Answer: Root Mean Squared Error (RMSE)
RMSE is the best choice because it penalizes larger errors more significantly due to the squaring of errors, and the final square root operation converts the error metric back into the original units of the target variable (dollars), making it more interpretable than MSE.
A data scientist trains a decision tree model. They observe that the model achieves 99% accuracy on the training data but only 70% accuracy on the test data. The performance on the training data is excellent, but the performance on the test data is significantly worse. This situation is a classic example of what?
Answer: High variance (Overfitting)
High variance, or overfitting, occurs when a model learns the training data too well, including its noise and random fluctuations, rather than the underlying general patterns. This results in excellent performance on the training data but poor performance on new, unseen data (the test set).
In the context of binary classification, which statement best describes what the Area Under the ROC Curve (AUC) represents?
Answer: The probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.
The AUC score is a measure of a classifier's ability to distinguish between classes. It is interpreted as the probability that the model assigns a higher score to a randomly chosen positive example than to a randomly chosen negative example. An AUC of 0.5 represents a model with no discrimination ability (random guessing), while an AUC of 1.0 represents a perfect classifier.
When building a machine learning model, a data scientist splits the data into three sets: training, validation, and test. What is the specific and primary role of the validation set?
Answer: To tune model hyperparameters and select the best performing model architecture before final evaluation.
The validation set is used to fine-tune the model's hyperparameters (e.g., learning rate, number of trees) and to compare different models. The model is trained on the training set, evaluated and tuned on the validation set, and then the final, chosen model is evaluated once on the test set to get an unbiased estimate of its performance on new data.