MAC MAC A/B Testing & Experimentation 2 — Questions and Answers
Question 1: What distinguishes a multivariate test (MVT) from a standard A/B test?
- MVT tests one element at a time; A/B tests multiple elements simultaneously
- MVT tests multiple elements and their combinations simultaneously; A/B tests one variable at a time (Correct answer)
- MVT requires no control group; A/B tests always require one
- MVT is used exclusively for email campaigns; A/B tests are for landing pages
Correct answer: MVT tests multiple elements and their combinations simultaneously; A/B tests one variable at a time
Multivariate testing simultaneously tests multiple page elements and their combinations to identify which combination produces the best outcome.
Question 2: In A/B testing, what is a 'false positive' (Type I error)?
- Concluding a variant is worse when it is actually better
- Concluding a variant is better when the difference is actually due to chance (Correct answer)
- Running a test on an insufficient sample size
- Failing to detect a real difference between variants
Correct answer: Concluding a variant is better when the difference is actually due to chance
A false positive (Type I error) occurs when you incorrectly conclude that a variant outperforms the control when the observed difference is actually random.
Question 3: What is 'traffic splitting' in the context of an A/B test?
- Routing users to different servers to reduce load
- Randomly assigning users to control or variation groups to ensure unbiased comparison (Correct answer)
- Segmenting test results by device type after the test concludes
- Distributing test results across multiple reporting dashboards
Correct answer: Randomly assigning users to control or variation groups to ensure unbiased comparison
Traffic splitting randomly divides incoming users between test variants so that each group is representative and results can be fairly compared.
Question 4: Why should an A/B test ideally run for at least one full business cycle (e.g., one week)?
- To give the testing platform time to initialize tracking
- To account for day-of-week traffic and behavior patterns that could skew results (Correct answer)
- To allow the variant to be indexed by search engines
- To collect enough impressions to trigger automated optimization
Correct answer: To account for day-of-week traffic and behavior patterns that could skew results
Running a test for at least a full week captures daily behavioral differences—such as weekday vs. weekend patterns—that could otherwise distort results.
Question 5: What is 'interaction effect' in multivariate testing?
- The combined impact of two or more simultaneously changed elements that differs from their individual effects (Correct answer)
- The effect of a test variant on users who interact with a chatbot
- The influence of personalization on test results
- The change in user behavior caused by the testing tool itself
Correct answer: The combined impact of two or more simultaneously changed elements that differs from their individual effects
An interaction effect occurs when the combined impact of changing two elements together is different from what would be predicted by looking at each element's effect individually.
Question 6: A marketing analyst runs an A/B test and finds a 10% lift in conversions with 97% confidence. What should the analyst do next?
- Immediately roll back the variant as the result is inconclusive
- Document the result, implement the winning variant, and plan follow-up tests (Correct answer)
- Continue running the test indefinitely to gather more data
- Rerun the test with a smaller traffic allocation
Correct answer: Document the result, implement the winning variant, and plan follow-up tests
A 97% confidence level exceeds the 95% threshold, making the result statistically significant—the analyst should implement the winner and document findings for future learning.
MAC MAC A/B Testing & Experimentation 2
What distinguishes a multivariate test (MVT) from a standard A/B test?