AML Quality Assurance & Improvement 2

Question 1

Which cross-validation strategy is most appropriate when your dataset has significant temporal ordering?

Accepted Answer

Time-series split (walk-forward)

Answer

Time-series split (walk-forward validation) prevents data leakage by always training on past data and validating on future data.

Question 2

A model achieves 99% accuracy on an imbalanced dataset where 99% of samples are class 0. What is the most important additional metric to evaluate?

Accepted Answer

Matthews Correlation Coefficient (MCC)

Answer

MCC accounts for all four confusion matrix values and provides a reliable metric even when classes are highly imbalanced.

Question 3

What does a calibration curve (reliability diagram) measure in a classifier?

Accepted Answer

Alignment between predicted probabilities and actual outcome frequencies

Answer

A calibration curve plots predicted probabilities against actual frequencies to show whether a model's confidence scores are trustworthy.

Question 4

You observe that validation loss stops improving after epoch 20 but training loss keeps decreasing. The correct QA response is to:

Accepted Answer

Apply early stopping at epoch 20 and regularize the model

Answer

The divergence between training and validation loss is a clear overfitting signal; early stopping and regularization directly address it.

Question 5

Which technique quantifies uncertainty by training multiple models on bootstrap samples of the training data?

Accepted Answer

Bootstrap aggregating (Bagging)

Answer

Bagging trains multiple models on bootstrapped datasets; the variance of their predictions estimates model uncertainty.

(AML) Advanced Machine Learning Certification Practice Test