Master of Data Science Research & Data Analysis 1 Flashcards

A data scientist splits a dataset into 80% training and 20% test sets, trains a complex neural network, and reports excellent test accuracy. A colleague suggests the model may still be overfit. Which additional evaluation strategy would best reveal this?

Answer: Use k-fold cross-validation on the full dataset and compare variance across folds

K-fold cross-validation assesses model stability across multiple train/test splits. High variance in scores across folds indicates overfitting that a single train/test split can miss, since that split may have been favorable by chance.

A researcher is building a predictive model on a highly imbalanced dataset where only 2% of records belong to the positive class. The model achieves 98% accuracy but fails to detect any positive cases. What is the most appropriate corrective step?

Answer: Use stratified sampling and evaluate with F1-score or AUC-ROC instead of accuracy

When classes are imbalanced, accuracy is a misleading metric since predicting the majority class always yields high accuracy. Stratified sampling preserves class distribution across splits, and metrics like F1-score or AUC-ROC properly account for performance on the minority class.

In a longitudinal study tracking student performance over four years, a researcher uses ordinary least squares regression and finds a statistically significant effect of tutoring hours. A reviewer raises concern about autocorrelation in the residuals. Which test should the researcher run first?

Answer: Durbin-Watson test

The Durbin-Watson test specifically detects autocorrelation in regression residuals. Autocorrelation is a common problem in time-series and longitudinal data because observations close in time tend to be correlated, violating the OLS independence assumption.

A master's student wants to understand which of 30 candidate features most strongly drive customer churn. She has limited domain knowledge and needs an interpretable ranking. Which approach is most appropriate as a first step?

Answer: Compute mutual information scores between each feature and the target variable

Mutual information measures the statistical dependency between each feature and the target without assuming a linear relationship, making it a model-agnostic and interpretable first-pass feature ranking. PCA creates new components rather than ranking original features, making it less useful for interpretability.

A data science researcher observes a strong positive correlation (r = 0.85) between ice cream sales and drowning rates across U.S. counties. What is the most accurate interpretation of this finding?

Answer: The correlation is spurious and likely driven by a confounding variable such as temperature or season

This is a classic example of confounding: warm weather increases both ice cream consumption and swimming activity, creating a spurious correlation between the two. Correlation alone never implies causation, and researchers must consider confounders before drawing causal conclusions.

A data science team is comparing two classification models using a held-out test set of 500 samples. Model A achieves 84% accuracy and Model B achieves 86% accuracy. Before concluding that Model B is superior, which statistical approach is most appropriate?

Answer: Perform McNemar's test on the models' correct/incorrect predictions for each sample

McNemar's test is specifically designed to compare two classifiers on the same test set by analyzing the contingency table of cases where the models disagree. It correctly accounts for the paired, binary nature of classification outcomes rather than treating predictions as continuous measurements.