AML Quality Assurance & Improvement 3

Question 1

Which statistical test is most appropriate for detecting covariate shift between training and production data distributions?

Accepted Answer

Two-sample Kolmogorov-Smirnov test

Answer

The two-sample KS test compares the empirical CDFs of two continuous distributions without assuming normality, making it ideal for detecting feature drift.

Question 2

Concept drift occurs when:

Accepted Answer

The relationship P(Y|X) between inputs and outputs changes over time

Answer

Concept drift specifically refers to a change in the conditional distribution P(Y|X), meaning the same inputs should now map to different outputs.

Question 3

A production model's average prediction score drops significantly with no change in input feature distributions. This most likely indicates:

Accepted Answer

A data pipeline serialization error

Answer

When input distributions are stable but prediction scores change, the issue is typically in data pipeline processing or feature engineering, not the model itself.

Question 4

Which monitoring metric is specifically designed to detect when a model's predictions become less correlated with eventual ground truth labels before those labels are available?

Accepted Answer

Population Stability Index (PSI)

Answer

PSI measures how much the distribution of model scores has shifted from a reference period, acting as an early proxy for performance degradation.

Question 5

In shadow mode deployment for ML QA, the shadow model's predictions are:

Accepted Answer

Logged and evaluated but never surfaced to end users

Answer

Shadow mode runs the candidate model in parallel, capturing predictions for offline evaluation without any user-facing impact.

(AML) Advanced Machine Learning Certification Practice Test

(AML) Advanced Machine Learning Certification Practice Test

AML Quality Assurance & Improvement 3