HAC Machine Learning in Healthcare 3 — Questions and Answers
Question 1: A convolutional neural network (CNN) trained to detect pneumonia on chest X-rays performs well on training data but poorly on external hospital data. This is best described as:
- Underfitting due to insufficient model complexity
- Dataset shift or distribution shift (Correct answer)
- Overfitting to training noise only
- Label leakage from the test set
Correct answer: Dataset shift or distribution shift
Dataset shift occurs when the statistical distribution of training data differs from deployment data, causing performance degradation on external cohorts.
Question 2: In survival analysis applied to patient outcomes, the Kaplan-Meier estimator is used to:
- Predict continuous lab values over time
- Estimate the probability of surviving past a given time point (Correct answer)
- Classify patients into risk quartiles using logistic regression
- Impute missing longitudinal data
Correct answer: Estimate the probability of surviving past a given time point
The Kaplan-Meier estimator produces a non-parametric survival curve showing the probability of event-free survival over time, accounting for censored observations.
Question 3: Which type of ML model is most commonly used for structured EHR tabular data in clinical risk scoring?
- Transformer (BERT-based)
- Convolutional Neural Network
- Gradient Boosted Trees (e.g., XGBoost) (Correct answer)
- Generative Adversarial Network
Correct answer: Gradient Boosted Trees (e.g., XGBoost)
Gradient boosted tree models like XGBoost consistently outperform deep learning on structured tabular EHR data and offer built-in feature importance.
Question 4: A healthcare ML model that denies prior authorization more frequently for a specific racial group is exhibiting:
- Selection bias in data collection
- Algorithmic bias causing disparate impact (Correct answer)
- Recall-precision tradeoff error
- Overfitting to demographic features
Correct answer: Algorithmic bias causing disparate impact
Algorithmic bias causes disparate impact when a model's outputs disproportionately disadvantage a protected group, often due to biased training data.
Question 5: When building a sepsis early-warning model, the AUROC metric measures:
- The time between prediction and clinical intervention
- The model's ability to discriminate sepsis from non-sepsis across all decision thresholds (Correct answer)
- The number of true positive predictions per hour
- The calibration between predicted and observed probabilities
Correct answer: The model's ability to discriminate sepsis from non-sepsis across all decision thresholds
AUROC (Area Under the ROC Curve) quantifies overall discriminative ability across all classification thresholds, with 1.0 being perfect discrimination.
Question 6: Which regularization technique is commonly used in neural networks to prevent overfitting by randomly deactivating neurons during training?
- L2 (Ridge) regularization
- Batch normalization
- Dropout (Correct answer)
- Early stopping only
Correct answer: Dropout
Dropout randomly sets a fraction of neuron activations to zero during each training step, forcing the network to learn redundant representations and reducing overfitting.
Question 7: In clinical decision support, a model's 'calibration' refers to:
- How fast the model produces predictions
- How well predicted probabilities match actual observed event rates (Correct answer)
- The number of features used in the model
- The model's performance on the training dataset
Correct answer: How well predicted probabilities match actual observed event rates
A well-calibrated model produces predicted probabilities that closely match the true frequency of outcomes (e.g., a 70% predicted risk should occur ~70% of the time).
HAC Machine Learning in Healthcare 3
A convolutional neural network (CNN) trained to detect pneumonia on chest X-rays performs well on training data but poorly on external hospital data.
This is best described as: