CAP Quality Assurance & Compliance 2 — Questions and Answers
Question 1: During model deployment, an analytics team discovers the production data pipeline delivers a feature in a different unit than the training data used. What quality assurance practice would most likely have caught this earlier?
- Data validation checks comparing training and production schemas (Correct answer)
- Increasing the model's regularization strength
- Running the model on a larger training dataset
- Adding more stakeholders to the sign-off meeting
Correct answer: Data validation checks comparing training and production schemas
Schema and data validation checks between training and production environments catch unit or format mismatches before deployment.
Question 2: A CAP-certified analyst is asked to reuse customer data collected for billing purposes in a new marketing propensity model. What compliance concern should be addressed first?
- Whether the marketing team has enough budget
- Whether the original consent and purpose of collection permits this new use (Correct answer)
- Whether the marketing model will have sufficient accuracy
- Whether the billing system can export the data quickly
Correct answer: Whether the original consent and purpose of collection permits this new use
Purpose limitation requires confirming that data collected for one purpose may lawfully and ethically be used for another.
Question 3: Which technique is most appropriate for verifying that an analytics model's outputs remain reliable after it has been in production for six months?
- Increasing the model's prediction threshold
- Archiving the original training dataset
- Ongoing performance monitoring against holdout or fresh labeled data (Correct answer)
- Re-running the original unit tests on the training code
Correct answer: Ongoing performance monitoring against holdout or fresh labeled data
Continuous monitoring against current labeled data detects drift and degradation that static tests cannot.
Question 4: An analyst finds that a predictive hiring model systematically scores applicants from one demographic group lower despite similar qualifications. Under professional ethics standards, what is the appropriate first step?
- Quietly adjust the scores for the affected group
- Remove the demographic variable and assume the problem is solved
- Continue using the model since it was already approved
- Document and report the disparity to stakeholders before continued use (Correct answer)
Correct answer: Document and report the disparity to stakeholders before continued use
Ethical practice requires transparently documenting and escalating potential bias rather than concealing or ignoring it.
Question 5: What is the primary purpose of maintaining an audit trail in an analytics project?
- To enable reconstruction and verification of how results were produced (Correct answer)
- To increase the processing speed of the analytics pipeline
- To reduce the amount of data storage required
- To eliminate the need for peer review
Correct answer: To enable reconstruction and verification of how results were produced
Audit trails record data sources, transformations, and decisions so results can be reproduced and verified.
Question 6: A financial services analytics team must comply with regulations requiring model explainability. Which model governance artifact best supports this requirement?
- The email thread approving the project budget
- Model documentation describing inputs, methodology, assumptions, and limitations (Correct answer)
- A compressed backup of the model's binary file
- A dashboard showing only the model's accuracy metric
Correct answer: Model documentation describing inputs, methodology, assumptions, and limitations
Comprehensive model documentation is the standard governance artifact demonstrating how and why a model makes decisions.
Question 7: During quality review, two analysts independently run the same analysis script on the same data and get different results. What is the most likely quality issue to investigate first?
- The sample size was too small
- The stakeholders changed their requirements
- Uncontrolled environment differences such as software versions or random seeds (Correct answer)
- The business question was poorly defined
Correct answer: Uncontrolled environment differences such as software versions or random seeds
Non-reproducible results from identical inputs usually stem from environment, dependency, or seed differences.
CAP Quality Assurance & Compliance 2
During model deployment, an analytics team discovers the production data pipeline delivers a feature in a different unit than the training data used.
What quality assurance practice would most likely have caught this earlier?