MS-DS Master of Data science Model Evaluation and Validation Questions and Answers

Question 1

A machine learning model is developed to detect a rare but critical disease. For this application, failing to identify a person who has the disease (a false negative) is a far more severe error than incorrectly flagging a healthy person (a false positive). Which evaluation metric should be prioritized for optimization?

Accepted Answer

Recall (Sensitivity)

Answer

Recall, also known as Sensitivity or True Positive Rate, measures the model's ability to find all actual positive cases (TP / (TP + FN)). In a critical medical diagnosis scenario, maximizing recall is crucial to minimize the number of dangerous false negatives.

Question 2

What is the primary advantage of using k-fold cross-validation over a single train-test split for evaluating a model's performance?

Accepted Answer

It provides a more robust and reliable estimate of the model's generalization performance by testing on multiple, different subsets of the data.

Answer

K-fold cross-validation provides a more stable estimate of a model's performance on unseen data. By training and testing the model on 'k' different subsets of the data and averaging the results, it reduces the variance associated with a single, potentially lucky or unlucky, train-test split.

Question 3

You are evaluating a regression model that predicts apartment rental prices in dollars. You need a metric that expresses the typical prediction error in dollars and penalizes larger errors more heavily. Which of the following metrics is the most suitable?

Accepted Answer

Root Mean Squared Error (RMSE)

Answer

RMSE is the best choice because it penalizes larger errors more significantly due to the squaring of errors, and the final square root operation converts the error metric back into the original units of the target variable (dollars), making it more interpretable than MSE.

Question 4

A data scientist trains a decision tree model. They observe that the model achieves 99% accuracy on the training data but only 70% accuracy on the test data. The performance on the training data is excellent, but the performance on the test data is significantly worse. This situation is a classic example of what?

Accepted Answer

High variance (Overfitting)

Answer

High variance, or overfitting, occurs when a model learns the training data too well, including its noise and random fluctuations, rather than the underlying general patterns. This results in excellent performance on the training data but poor performance on new, unseen data (the test set).

Question 5

In the context of binary classification, which statement best describes what the Area Under the ROC Curve (AUC) represents?

Accepted Answer

The probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

Answer

The AUC score is a measure of a classifier's ability to distinguish between classes. It is interpreted as the probability that the model assigns a higher score to a randomly chosen positive example than to a randomly chosen negative example. An AUC of 0.5 represents a model with no discrimination ability (random guessing), while an AUC of 1.0 represents a perfect classifier.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Model Evaluation and Validation Questions and Answers