MS-DS Master of Data science Master of Data Science Research & Data Analysis 1 — Questions and Answers
Question 1: A data scientist splits a dataset into 80% training and 20% test sets, trains a complex neural network, and reports excellent test accuracy. A colleague suggests the model may still be overfit. Which additional evaluation strategy would best reveal this?
- Increase the test set size to 40% and retrain
- Use k-fold cross-validation on the full dataset and compare variance across folds (Correct answer)
- Apply dimensionality reduction before retraining
- Report precision and recall instead of accuracy
Correct answer: Use k-fold cross-validation on the full dataset and compare variance across folds
K-fold cross-validation assesses model stability across multiple train/test splits. High variance in scores across folds indicates overfitting that a single train/test split can miss, since that split may have been favorable by chance.
Question 2: A researcher is building a predictive model on a highly imbalanced dataset where only 2% of records belong to the positive class. The model achieves 98% accuracy but fails to detect any positive cases. What is the most appropriate corrective step?
- Increase the learning rate of the model
- Switch from a classification model to a regression model
- Use stratified sampling and evaluate with F1-score or AUC-ROC instead of accuracy (Correct answer)
- Remove negative class samples until classes are equal
Correct answer: Use stratified sampling and evaluate with F1-score or AUC-ROC instead of accuracy
When classes are imbalanced, accuracy is a misleading metric since predicting the majority class always yields high accuracy. Stratified sampling preserves class distribution across splits, and metrics like F1-score or AUC-ROC properly account for performance on the minority class.
Question 3: In a longitudinal study tracking student performance over four years, a researcher uses ordinary least squares regression and finds a statistically significant effect of tutoring hours. A reviewer raises concern about autocorrelation in the residuals. Which test should the researcher run first?
- Shapiro-Wilk test
- Levene's test
- Durbin-Watson test (Correct answer)
- Breusch-Pagan test
Correct answer: Durbin-Watson test
The Durbin-Watson test specifically detects autocorrelation in regression residuals. Autocorrelation is a common problem in time-series and longitudinal data because observations close in time tend to be correlated, violating the OLS independence assumption.
Question 4: A master's student wants to understand which of 30 candidate features most strongly drive customer churn. She has limited domain knowledge and needs an interpretable ranking. Which approach is most appropriate as a first step?
- Train a deep neural network and inspect the final layer weights
- Compute mutual information scores between each feature and the target variable (Correct answer)
- Apply PCA and select the top 5 principal components
- Use forward stepwise regression with AIC as the stopping criterion
Correct answer: Compute mutual information scores between each feature and the target variable
Mutual information measures the statistical dependency between each feature and the target without assuming a linear relationship, making it a model-agnostic and interpretable first-pass feature ranking. PCA creates new components rather than ranking original features, making it less useful for interpretability.
Question 5: A data science researcher observes a strong positive correlation (r = 0.85) between ice cream sales and drowning rates across U.S. counties. What is the most accurate interpretation of this finding?
- Ice cream consumption causally increases drowning risk and should be studied further with a randomized trial
- The correlation is spurious and likely driven by a confounding variable such as temperature or season (Correct answer)
- The Pearson correlation coefficient is too high to be trustworthy and data collection should be repeated
- A regression model should be built immediately to quantify the causal effect
Correct answer: The correlation is spurious and likely driven by a confounding variable such as temperature or season
This is a classic example of confounding: warm weather increases both ice cream consumption and swimming activity, creating a spurious correlation between the two. Correlation alone never implies causation, and researchers must consider confounders before drawing causal conclusions.
Question 6: A data science team is comparing two classification models using a held-out test set of 500 samples. Model A achieves 84% accuracy and Model B achieves 86% accuracy. Before concluding that Model B is superior, which statistical approach is most appropriate?
- Accept Model B since 2 percentage points is always a meaningful difference in practice
- Run a paired t-test on the raw prediction vectors from both models
- Perform McNemar's test on the models' correct/incorrect predictions for each sample (Correct answer)
- Compute the standard deviation of each model's accuracy using bootstrap resampling only
Correct answer: Perform McNemar's test on the models' correct/incorrect predictions for each sample
McNemar's test is specifically designed to compare two classifiers on the same test set by analyzing the contingency table of cases where the models disagree. It correctly accounts for the paired, binary nature of classification outcomes rather than treating predictions as continuous measurements.
MS-DS Master of Data science Master of Data Science Research & Data Analysis 1
A data scientist splits a dataset into 80% training and 20% test sets, trains a complex neural network, and reports excellent test accuracy.
A colleague suggests the model may still be overfit.
Which additional evaluation strategy would best reveal this?