Conversion Optimization A/B Testing and Experimentation 2 — Questions and Answers
Question 1: What is multivariate testing (MVT) and how does it differ from A/B testing?
- MVT tests multiple element combinations simultaneously, while A/B tests one change at a time (Correct answer)
- MVT tests mobile vs desktop, while A/B tests only desktop
- MVT uses heatmaps, while A/B uses click tracking
- MVT is faster to run than A/B testing
Correct answer: MVT tests multiple element combinations simultaneously, while A/B tests one change at a time
Multivariate testing evaluates multiple combinations of elements at once to find the best-performing combination, requiring more traffic than A/B tests.
Question 2: What minimum sample size principle ensures reliable A/B test results?
- Reach statistical power (typically 80%) before concluding a test (Correct answer)
- Run the test for exactly 7 days regardless of traffic
- Collect at least 100 conversions per variant
- Stop the test as soon as one variant leads
Correct answer: Reach statistical power (typically 80%) before concluding a test
Achieving 80% statistical power ensures the test has enough sample size to detect a true difference if one exists.
Question 3: What is 'peeking' in A/B testing and why is it problematic?
- Stopping a test early after seeing an early winner, which inflates false-positive rates (Correct answer)
- Viewing competitor test results
- Running tests without user consent
- Duplicating test variants
Correct answer: Stopping a test early after seeing an early winner, which inflates false-positive rates
Peeking causes inflated false-positive rates because early results fluctuate and are not yet statistically stable.
Question 4: Which type of A/B test traffic split is most common for standard experiments?
- 50/50 split between control and variation (Correct answer)
- 90/10 split favoring the control
- 25/25/25/25 for four variants
- 70/30 split always favoring the new variant
Correct answer: 50/50 split between control and variation
A 50/50 split is the most efficient allocation for detecting differences between two variants with equal precision.
Question 5: What does 'uplift' mean in A/B testing results?
- The percentage improvement of the variation over the control (Correct answer)
- The increase in website traffic during the test
- The number of new users acquired
- The reduction in page load time achieved
Correct answer: The percentage improvement of the variation over the control
Uplift quantifies how much better the variation performed compared to the control, expressed as a percentage.
Question 6: Why is it important to run A/B tests for full business cycles (e.g., full weeks)?
- To account for day-of-week behavioral patterns that could skew results (Correct answer)
- To satisfy legal testing duration requirements
- To allow search engines to index both variants
- To reduce CDN caching issues during the test
Correct answer: To account for day-of-week behavioral patterns that could skew results
User behavior varies by day of week, so running tests for complete weeks prevents weekday/weekend traffic patterns from distorting the results.
Conversion Optimization A/B Testing and Experimentation 2
What is multivariate testing (MVT) and how does it differ from A/B testing?