Free Data Science Model Evaluation and Validation Questions and Answers 1

Question 1

A data scientist is developing a binary classification model to detect a rare but life-threatening disease. The consequences of failing to identify a sick patient (a false negative) are far more severe than incorrectly classifying a healthy patient as sick (a false positive). Which evaluation metric should be prioritized for optimization?

Accepted Answer

Recall (Sensitivity)

Answer

Recall (or Sensitivity) measures the model's ability to correctly identify all actual positive cases (TP / (TP + FN)). In this medical scenario, minimizing false negatives (FN) is critical, which is precisely what maximizing recall achieves. Precision focuses on minimizing false positives, and accuracy can be misleading with imbalanced datasets.

Question 2

What is the primary advantage of using K-Fold Cross-Validation compared to a single train-test split for model evaluation?

Accepted Answer

It provides a more robust estimate of the model's performance on unseen data.

Answer

K-Fold Cross-Validation provides a more reliable and less biased estimate of model performance by training and evaluating the model on multiple, different subsets of the data. A single train-test split's result can be highly dependent on which specific data points happen to end up in the training vs. test set, making it less robust.

Question 3

A real estate company has built a model to predict house prices. They want to evaluate the model's performance by measuring the average absolute difference between the predicted prices and the actual prices, in the original currency (e.g., dollars). Which metric is most suitable for this purpose?

Accepted Answer

Mean Absolute Error (MAE)

Answer

Mean Absolute Error (MAE) calculates the average of the absolute differences between predicted and actual values. This gives a straightforward, interpretable measure of the average error magnitude in the original units of the target variable (in this case, dollars). MSE and RMSE square the errors, which penalizes larger errors more heavily and results in units that are squared (MSE) or back in the original units but influenced by the squaring (RMSE). R-squared measures the proportion of variance explained, not the error in original units.

Question 4

Which of the following best describes a model that is suffering from high bias?

Accepted Answer

The model is too simple and fails to capture the underlying patterns in both the training and test data.

Answer

High bias is synonymous with underfitting. An underfit model is too simplistic and makes strong assumptions about the data, leading it to perform poorly on both the training set and the test set because it cannot capture the true underlying relationships. High variance, in contrast, describes an overfit model that captures noise and performs well on training data but poorly on test data.

Question 5

A data science team is tuning a model with a very large hyperparameter space and has limited computational resources. They need an efficient method to find a good combination of hyperparameters without exhaustively testing every possibility. Which approach is generally more efficient than Grid Search in this scenario?

Accepted Answer

Random Search

Answer

Random Search is often more computationally efficient than Grid Search when dealing with a large hyperparameter space. It works by sampling a fixed number of random combinations from the specified parameter distributions. Research has shown that Random Search is more likely to discover good values for important hyperparameters compared to Grid Search, which spends too much time exploring unimportant dimensions.

Data Science Practice Test

Data Science Practice Test

Free Data Science Model Evaluation and Validation Questions and Answers 1