AML Risk Assessment & Management 3 — Questions and Answers
Question 1: Which metric is most useful for risk assessment in a fraud detection model where false negatives are far more costly than false positives?
- Accuracy
- Recall (Sensitivity) (Correct answer)
- Specificity
- R-squared
Correct answer: Recall (Sensitivity)
Recall measures the proportion of actual fraud cases correctly identified, directly capturing the cost of missing true fraudulent transactions.
Question 2: A model's Expected Calibration Error (ECE) is high. What risk does this introduce in a decision-support system?
- The model trains too slowly for production use
- Predicted probabilities misrepresent true likelihoods, leading to misinformed decisions (Correct answer)
- Feature importances become unstable across runs
- The model cannot handle missing input values
Correct answer: Predicted probabilities misrepresent true likelihoods, leading to misinformed decisions
High ECE means confidence scores are unreliable proxies for actual outcome probabilities, so decision thresholds calibrated on them will systematically under- or over-estimate real risk.
Question 3: What is 'concept drift' and why is it a risk management concern in deployed ML systems?
- Gradual memory leak in model serving infrastructure
- A change in the statistical relationship between inputs and the target variable over time (Correct answer)
- Random variation in model outputs due to floating-point precision
- Instability in gradient descent caused by poor initialization
Correct answer: A change in the statistical relationship between inputs and the target variable over time
Concept drift occurs when the underlying data-generating process changes so that previously learned input-output relationships no longer hold, degrading model reliability.
Question 4: Which of the following best represents a proactive risk control in an ML model lifecycle?
- Retraining the model after a production incident occurs
- Implementing automated performance monitoring with alerting thresholds before deployment (Correct answer)
- Documenting model failures in post-mortems
- Rolling back the model version after detecting degradation
Correct answer: Implementing automated performance monitoring with alerting thresholds before deployment
Proactive controls prevent or detect problems before they cause harm; automated monitoring with pre-set alert thresholds catches degradation before it escalates to an incident.
Question 5: When performing a risk assessment for an ML model, 'inherent risk' refers to:
- Residual risk remaining after controls are applied
- The raw level of risk before any mitigation controls are in place (Correct answer)
- Risk introduced by third-party data vendors
- The risk of overfitting to the validation set
Correct answer: The raw level of risk before any mitigation controls are in place
Inherent risk is the exposure to adverse outcomes in the absence of any risk controls, providing a baseline for understanding how much mitigation is needed.
Question 6: Which approach best supports 'right to explanation' requirements while managing the risk of opaque ML decisions in regulated sectors?
- Using deeper neural networks to improve accuracy
- Applying post-hoc explainability methods such as SHAP or LIME (Correct answer)
- Increasing training data volume without feature engineering
- Switching to unsupervised learning to remove labels
Correct answer: Applying post-hoc explainability methods such as SHAP or LIME
SHAP and LIME generate feature-level explanations for individual predictions, enabling organizations to justify automated decisions to regulators and affected individuals.
Question 7: A risk review board requires a 'challenger model' alongside the production ('champion') model. What risk management purpose does this serve?
- It reduces training time by distributing computation
- It provides a ready alternative to compare against and replace the champion if performance degrades (Correct answer)
- It generates synthetic data to augment the training set
- It handles outlier inputs that the champion model rejects
Correct answer: It provides a ready alternative to compare against and replace the champion if performance degrades
The champion-challenger framework maintains a validated alternative model in parallel so performance comparisons are continuous and a replacement is available without emergency development.
AML Risk Assessment & Management 3
Which metric is most useful for risk assessment in a fraud detection model where false negatives are far more costly than false positives?