MAC Risk Assessment & Mitigation 3 — Questions and Answers
Question 1: A brand launches a new ad campaign and sees a spike in direct traffic the following week. The analytics team attributes all direct traffic growth to brand awareness. What risk does this interpretation carry?
- Confirmation bias leading to over-crediting the campaign without ruling out alternative causes (Correct answer)
- Simpson's paradox caused by aggregating across market segments
- Regression to the mean inflating perceived campaign impact
- Multicollinearity reducing model coefficient stability
Correct answer: Confirmation bias leading to over-crediting the campaign without ruling out alternative causes
Attributing all direct traffic growth to the campaign without controlling for seasonality or other concurrent activities is confirmation bias.
Question 2: Which approach best mitigates the risk of campaign decisions being made on statistically insignificant A/B test results?
- Running tests for a fixed two-week window regardless of traffic volume
- Pre-specifying minimum detectable effect size and required sample size before launching the test (Correct answer)
- Stopping the test when one variant reaches a 55% win rate
- Using a one-tailed test to reduce the p-value threshold
Correct answer: Pre-specifying minimum detectable effect size and required sample size before launching the test
Pre-specifying MDE and sample size ensures the test is adequately powered before launch, preventing premature conclusions from underpowered results.
Question 3: A media mix model (MMM) trained on three years of data is used to allocate next year's budget. Model performance degrades significantly in the second quarter. Which risk is MOST likely responsible?
- Heteroscedasticity in the residuals
- Model staleness due to changed market dynamics post-training period (Correct answer)
- Overfitting caused by too many predictor variables
- Autocorrelation in weekly spend data
Correct answer: Model staleness due to changed market dynamics post-training period
MMMs trained on historical data degrade when market conditions, competitive activity, or consumer behavior shift, a form of model drift.
Question 4: An analyst uses the same dataset to both build and evaluate a predictive churn model. What data risk does this create?
- Data leakage causing overly optimistic performance estimates (Correct answer)
- Class imbalance leading to biased precision scores
- Feature multicollinearity inflating variable importance scores
- Survivorship bias excluding churned customers from training
Correct answer: Data leakage causing overly optimistic performance estimates
Evaluating a model on its training data inflates performance metrics because the model has already 'seen' those examples.
Question 5: A marketing team relies on third-party cookie data for audience segmentation. Which strategic risk is MOST critical to address given current browser privacy changes?
- Dependency on a data source that is being deprecated, threatening segmentation accuracy (Correct answer)
- Increased cost-per-click as cookie pools shrink
- Lower open rates in email campaigns due to cookie blocking
- Reduced social media reach from cookie-gated lookalike audiences
Correct answer: Dependency on a data source that is being deprecated, threatening segmentation accuracy
Third-party cookie deprecation directly eliminates the data source the segmentation relies on, requiring migration to first-party or cookieless alternatives.
Question 6: Which scenario represents a data quality risk caused by schema drift in a marketing data pipeline?
- An upstream CRM adds a new field format, breaking downstream joins and causing null values in reports (Correct answer)
- A dashboard loads slowly because of unindexed database queries
- Campaign UTM parameters are inconsistently applied across channels
- An analyst accidentally deletes a key report table
Correct answer: An upstream CRM adds a new field format, breaking downstream joins and causing null values in reports
Schema drift occurs when upstream data structure changes unexpectedly, breaking downstream transformations and introducing silent data errors.
Question 7: A DTC brand's conversion rate model shows high accuracy in backtests but fails to predict sales during a flash sale event. Which mitigation would BEST address this for future models?
- Incorporating promotional event flags as a feature in the model training data (Correct answer)
- Switching from logistic regression to a neural network architecture
- Increasing training data recency by using only the last 90 days
- Applying SMOTE to balance conversion and non-conversion classes
Correct answer: Incorporating promotional event flags as a feature in the model training data
Including a promotional event indicator variable allows the model to account for the behavioral shift that occurs during sales, reducing forecast error.
MAC Risk Assessment & Mitigation 3
A brand launches a new ad campaign and sees a spike in direct traffic the following week.
The analytics team attributes all direct traffic growth to brand awareness.
What risk does this interpretation carry?