MAC A/B Testing & Experimentation Flashcards
6 cards from real MAC practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.
Read the first 6 MAC A/B Testing & Experimentation flashcards as text
What is 'personalization' in digital marketing experimentation and how does it differ from A/B testing?
Answer: Personalization tailors content to specific user segments; A/B testing randomly splits users to find a universally better variant
Personalization delivers targeted experiences to defined user segments based on their attributes, while A/B testing finds the single best variation for the overall audience.
What is a 'holdout group' in the context of marketing experimentation?
Answer: A segment of users deliberately excluded from all test variants to measure the baseline effect over time
A holdout group is a subset of users kept out of all experiments so analysts can measure the cumulative incremental impact of ongoing optimizations over time.
In email marketing A/B testing, which element test is most commonly used to improve open rates?
Answer: Email subject line
Subject line tests are the most widely used email A/B tests because the subject line is the primary factor that determines whether a recipient opens an email.
What is 'Bayesian A/B testing' and how does it differ from the traditional frequentist approach?
Answer: Bayesian testing uses historical data to update the probability of a variant winning in real time; frequentist testing uses a fixed significance threshold at the end
Bayesian testing continuously updates the probability that a variant is the winner as data accumulates, whereas frequentist testing evaluates significance at a predetermined endpoint.
What is the risk of running too many simultaneous A/B tests on the same web page?
Answer: Interaction effects between tests can corrupt results and make it impossible to isolate the impact of any single change
Running multiple overlapping tests on the same page creates interaction effects where changes from one test influence the results of another, making it impossible to attribute outcomes accurately.
When documenting A/B test results for a stakeholder report, which element is most important to include alongside the winning variant?
Answer: The hypothesis tested, the confidence level, sample size, and the measured lift
A complete test report should document the hypothesis, confidence level, sample size, and lift so stakeholders can evaluate the reliability of the result and apply the learning to future decisions.