AML Quality Assurance & Improvement 3 — Questions and Answers
Question 1: Which statistical test is most appropriate for detecting covariate shift between training and production data distributions?
- Paired t-test
- Two-sample Kolmogorov-Smirnov test (Correct answer)
- Chi-squared goodness-of-fit test
- ANOVA
Correct answer: Two-sample Kolmogorov-Smirnov test
The two-sample KS test compares the empirical CDFs of two continuous distributions without assuming normality, making it ideal for detecting feature drift.
Question 2: Concept drift occurs when:
- The input feature distribution P(X) changes over time
- The relationship P(Y|X) between inputs and outputs changes over time (Correct answer)
- The model's weights decay due to long deployment
- Training batch size becomes inconsistent
Correct answer: The relationship P(Y|X) between inputs and outputs changes over time
Concept drift specifically refers to a change in the conditional distribution P(Y|X), meaning the same inputs should now map to different outputs.
Question 3: A production model's average prediction score drops significantly with no change in input feature distributions. This most likely indicates:
- Covariate shift in upstream features
- Label shift or concept drift in the output distribution
- A data pipeline serialization error (Correct answer)
- Overfitting during the last training run
Correct answer: A data pipeline serialization error
When input distributions are stable but prediction scores change, the issue is typically in data pipeline processing or feature engineering, not the model itself.
Question 4: Which monitoring metric is specifically designed to detect when a model's predictions become less correlated with eventual ground truth labels before those labels are available?
- Population Stability Index (PSI) (Correct answer)
- Prediction drift score
- Shadow mode evaluation
- Wasserstein distance on output distribution
Correct answer: Population Stability Index (PSI)
PSI measures how much the distribution of model scores has shifted from a reference period, acting as an early proxy for performance degradation.
Question 5: In shadow mode deployment for ML QA, the shadow model's predictions are:
- Served to a random 50% of users
- Logged and evaluated but never surfaced to end users (Correct answer)
- Used to override the primary model when confidence is higher
- Deployed only on non-critical traffic paths
Correct answer: Logged and evaluated but never surfaced to end users
Shadow mode runs the candidate model in parallel, capturing predictions for offline evaluation without any user-facing impact.
Question 6: When setting up automated model retraining triggers in a production ML system, which signal is most directly actionable?
- Increase in raw data volume
- Statistically significant drop in held-out evaluation metric (Correct answer)
- Growth in number of API requests
- Increase in model inference latency
Correct answer: Statistically significant drop in held-out evaluation metric
A statistically significant performance degradation on a labeled evaluation set directly indicates the model needs retraining with updated data.
Question 7: What is the primary risk of using a static holdout test set for ongoing model quality monitoring over multiple retraining cycles?
- Overfitting to the validation set through hyperparameter tuning
- Test set contamination through repeated model selection on the same data (Correct answer)
- Increased inference cost per evaluation cycle
- Instability in precision-recall tradeoffs
Correct answer: Test set contamination through repeated model selection on the same data
Repeated selection of models based on the same test set causes the test set to implicitly influence model development, leaking information and inflating performance estimates.
AML Quality Assurance & Improvement 3
Which statistical test is most appropriate for detecting covariate shift between training and production data distributions?