Conversion Optimization A/B Testing and Experimentation 2

Question 1

What is multivariate testing (MVT) and how does it differ from A/B testing?

Accepted Answer

MVT tests multiple element combinations simultaneously, while A/B tests one change at a time

Answer

Multivariate testing evaluates multiple combinations of elements at once to find the best-performing combination, requiring more traffic than A/B tests.

Question 2

What minimum sample size principle ensures reliable A/B test results?

Accepted Answer

Reach statistical power (typically 80%) before concluding a test

Answer

Achieving 80% statistical power ensures the test has enough sample size to detect a true difference if one exists.

Question 3

What is 'peeking' in A/B testing and why is it problematic?

Accepted Answer

Stopping a test early after seeing an early winner, which inflates false-positive rates

Answer

Peeking causes inflated false-positive rates because early results fluctuate and are not yet statistically stable.

Question 4

Which type of A/B test traffic split is most common for standard experiments?

Accepted Answer

50/50 split between control and variation

Answer

A 50/50 split is the most efficient allocation for detecting differences between two variants with equal precision.

Question 5

What does 'uplift' mean in A/B testing results?

Accepted Answer

The percentage improvement of the variation over the control

Answer

Uplift quantifies how much better the variation performed compared to the control, expressed as a percentage.

Conversion Optimization Practice Test

Conversion Optimization Practice Test

Conversion Optimization A/B Testing and Experimentation 2