MAC MAC A/B Testing & Experimentation

Question 1

In a marketing A/B test, what does the 'control' represent?

Accepted Answer

The original version against which variations are compared

Answer

The new variation being tested

Answer

The statistical significance threshold

Answer

The holdout group excluded from all variants

Question 2

What does statistical significance indicate in the context of an A/B test?

Accepted Answer

The observed difference between variants is unlikely to be due to chance

Answer

The test has reached the required sample size

Answer

The winning variant has been deployed to all users

Answer

The test has run for a minimum of 30 days

Question 3

Why is it important to define a primary metric before launching an A/B test?

Accepted Answer

To prevent 'p-hacking' by changing the success metric after seeing results

Answer

To limit the number of variants that can be tested

Answer

To ensure the test runs for exactly two weeks

Answer

To automatically allocate traffic between variants

Question 4

What is the minimum recommended statistical confidence level commonly used for marketing A/B tests?

Accepted Answer

95%

Answer

75%

Answer

80%

Answer

99.9%

Question 5

What is 'sample size' in the context of A/B testing, and why does it matter?

Accepted Answer

The number of users in each group; too few users leads to unreliable results

Answer

The number of variants being tested; more variants means faster results

Answer

The length of time a test runs; longer tests are always more accurate

Answer

The percentage of traffic excluded from the test

Question 6

What is 'novelty effect' and how can it distort A/B test results?

Accepted Answer

Users clicking on a new variant simply because it is different, inflating its early performance

Answer

A bug that causes the new variant to load faster than the control

Answer

Statistical bias introduced by running tests during holiday periods

Answer

The tendency for returning users to prefer the control version