MAC MAC A/B Testing & Experimentation

Question 1

In a marketing A/B test, what does the 'control' represent?

Accepted Answer

The original version against which variations are compared

Answer

The control is the original, unchanged version of a page or element that serves as the baseline for comparing the performance of new variations.

Question 2

What does statistical significance indicate in the context of an A/B test?

Accepted Answer

The observed difference between variants is unlikely to be due to chance

Answer

Statistical significance indicates there is a sufficiently low probability that the measured difference between variants occurred by random chance.

Question 3

Why is it important to define a primary metric before launching an A/B test?

Accepted Answer

To prevent 'p-hacking' by changing the success metric after seeing results

Answer

Pre-defining the primary metric prevents cherry-picking results after the test, which inflates the false positive rate and leads to unreliable conclusions.

Question 4

What is the minimum recommended statistical confidence level commonly used for marketing A/B tests?

Accepted Answer

95%

Answer

A 95% confidence level (p-value ≤ 0.05) is the widely accepted standard for declaring a marketing test result statistically significant.

Question 5

What is 'sample size' in the context of A/B testing, and why does it matter?

Accepted Answer

The number of users in each group; too few users leads to unreliable results

Answer

Sample size refers to the number of users in each test group; insufficient sample sizes produce unreliable results because small differences can appear significant by chance.

(MAC) Marketing Analytics Certification Practice Test

(MAC) Marketing Analytics Certification Practice Test

MAC MAC A/B Testing & Experimentation