Conversion Optimization A/B Testing and Experimentation 2

Question 1

What is multivariate testing (MVT) and how does it differ from A/B testing?

Accepted Answer

MVT tests multiple element combinations simultaneously, while A/B tests one change at a time

Answer

MVT tests mobile vs desktop, while A/B tests only desktop

Answer

MVT uses heatmaps, while A/B uses click tracking

Answer

MVT is faster to run than A/B testing

Question 2

What minimum sample size principle ensures reliable A/B test results?

Accepted Answer

Reach statistical power (typically 80%) before concluding a test

Answer

Run the test for exactly 7 days regardless of traffic

Answer

Collect at least 100 conversions per variant

Answer

Stop the test as soon as one variant leads

Question 3

What is 'peeking' in A/B testing and why is it problematic?

Accepted Answer

Stopping a test early after seeing an early winner, which inflates false-positive rates

Answer

Viewing competitor test results

Answer

Running tests without user consent

Answer

Duplicating test variants

Question 4

Which type of A/B test traffic split is most common for standard experiments?

Accepted Answer

50/50 split between control and variation

Answer

90/10 split favoring the control

Answer

25/25/25/25 for four variants

Answer

70/30 split always favoring the new variant

Question 5

What does 'uplift' mean in A/B testing results?

Accepted Answer

The percentage improvement of the variation over the control

Answer

The increase in website traffic during the test

Answer

The number of new users acquired

Answer

The reduction in page load time achieved

Question 6

Why is it important to run A/B tests for full business cycles (e.g., full weeks)?

Accepted Answer

To account for day-of-week behavioral patterns that could skew results

Answer

To satisfy legal testing duration requirements

Answer

To allow search engines to index both variants

Answer

To reduce CDN caching issues during the test