MS-DS Master of Data science Model Evaluation and Validation Questions and Answers

Question 1

A machine learning model is developed to detect a rare but critical disease.
For this application, failing to identify a person who has the disease (a false negative) is a far more severe error than incorrectly flagging a healthy person (a false positive).
Which evaluation metric should be prioritized for optimization?

Accepted Answer

Recall (Sensitivity)

Answer

Accuracy

Answer

Precision

Answer

F1-Score

Question 2

What is the primary advantage of using k-fold cross-validation over a single train-test split for evaluating a model's performance?

Accepted Answer

It provides a more robust and reliable estimate of the model's generalization performance by testing on multiple, different subsets of the data.

Answer

It significantly reduces the computational time required to train the model since each 'fold' is smaller than the full training set.

Answer

It completely eliminates the possibility of overfitting during the training process.

Answer

It automatically selects the best hyperparameters for the model without needing a separate validation set.

Question 3

You are evaluating a regression model that predicts apartment rental prices in dollars. You need a metric that expresses the typical prediction error in dollars and penalizes larger errors more heavily. Which of the following metrics is the most suitable?

Accepted Answer

Root Mean Squared Error (RMSE)

Answer

Mean Absolute Error (MAE)

Answer

R-squared (R²)

Answer

Mean Squared Error (MSE)

Question 4

A data scientist trains a decision tree model. They observe that the model achieves 99% accuracy on the training data but only 70% accuracy on the test data. The performance on the training data is excellent, but the performance on the test data is significantly worse. This situation is a classic example of what?

Accepted Answer

High variance (Overfitting)

Answer

Data leakage

Answer

High bias (Underfitting)

Answer

A well-generalized model

Question 5

In the context of binary classification, which statement best describes what the Area Under the ROC Curve (AUC) represents?

Accepted Answer

The probability that the model will rank a randomly chosen positive instance higher than a randomly chosen negative instance.

Answer

The model's overall accuracy at a default classification threshold of 0.5.

Answer

The trade-off between the model's precision and recall.

Answer

The total number of correct predictions made by the model.

Question 6

When building a machine learning model, a data scientist splits the data into three sets: training, validation, and test. What is the specific and primary role of the validation set?

Accepted Answer

To tune model hyperparameters and select the best performing model architecture before final evaluation.

Answer

To provide a final, unbiased assessment of the selected model's performance on completely unseen data.

Answer

To be used for the primary training of the model's parameters (e.g., weights or coefficients).

Answer

To check for and remove outliers and anomalies from the original dataset.