AML Data Analysis & Reporting 5

Question 1

What is the purpose of cross-validation in model evaluation reporting?

Accepted Answer

To provide an unbiased estimate of model performance on unseen data

Answer

Cross-validation estimates generalization performance by rotating held-out folds, reducing reliance on a single train-test split that may be lucky or unlucky.

Question 2

A Kolmogorov-Smirnov (KS) test in credit scoring reports the maximum separation between which two curves?

Accepted Answer

Cumulative distribution functions of good and bad borrowers

Answer

The KS statistic measures the maximum vertical distance between the CDFs of positive and negative classes, quantifying a model's discriminatory power.

Question 3

When reporting model drift in production, which type of drift refers to changes in the input feature distribution?

Accepted Answer

Covariate shift

Answer

Covariate shift occurs when the marginal distribution of input features P(X) changes while the conditional P(Y|X) remains stable.

Question 4

In cohort analysis, what does retention rate measure?

Accepted Answer

The fraction of users from an initial cohort who remain active in a later period

Answer

Retention rate tracks what fraction of users who joined in a given cohort period are still active N periods later.

Question 5

Which pandas function is most efficient for computing group-level summary statistics across a large DataFrame?

Accepted Answer

df.groupby().agg()

Answer

groupby().agg() leverages vectorized operations internally, making it far more efficient than row-wise apply or iterrows for group aggregations.

(AML) Advanced Machine Learning Certification Practice Test

(AML) Advanced Machine Learning Certification Practice Test

AML Data Analysis & Reporting 5