MAC A/B Testing & Experimentation Flashcards
6 cards from real MAC practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.
Read the first 6 MAC A/B Testing & Experimentation flashcards as text
What distinguishes a multivariate test (MVT) from a standard A/B test?
Answer: MVT tests multiple elements and their combinations simultaneously; A/B tests one variable at a time
Multivariate testing simultaneously tests multiple page elements and their combinations to identify which combination produces the best outcome.
In A/B testing, what is a 'false positive' (Type I error)?
Answer: Concluding a variant is better when the difference is actually due to chance
A false positive (Type I error) occurs when you incorrectly conclude that a variant outperforms the control when the observed difference is actually random.
What is 'traffic splitting' in the context of an A/B test?
Answer: Randomly assigning users to control or variation groups to ensure unbiased comparison
Traffic splitting randomly divides incoming users between test variants so that each group is representative and results can be fairly compared.
Why should an A/B test ideally run for at least one full business cycle (e.g., one week)?
Answer: To account for day-of-week traffic and behavior patterns that could skew results
Running a test for at least a full week captures daily behavioral differences—such as weekday vs. weekend patterns—that could otherwise distort results.
What is 'interaction effect' in multivariate testing?
Answer: The combined impact of two or more simultaneously changed elements that differs from their individual effects
An interaction effect occurs when the combined impact of changing two elements together is different from what would be predicted by looking at each element's effect individually.
A marketing analyst runs an A/B test and finds a 10% lift in conversions with 97% confidence. What should the analyst do next?
Answer: Document the result, implement the winning variant, and plan follow-up tests
A 97% confidence level exceeds the 95% threshold, making the result statistically significant—the analyst should implement the winner and document findings for future learning.