CWA Conversion Rate Optimization & A/B Testing Flashcards

What is a 'hypothesis' in A/B testing?

Answer: A testable prediction about how a change will affect user behavior

An A/B test hypothesis is a structured, testable prediction stating that a specific change will produce a measurable improvement in a defined metric.

What is 'multivariate testing' (MVT) in CRO?

Answer: Testing multiple changes across multiple page elements at the same time

Multivariate testing simultaneously tests multiple variations of multiple page elements to identify which combination produces the best results.

What is the 'control' in an A/B test?

Answer: The original, unchanged version of the page

The control is the existing, unmodified version of the page that serves as the baseline against which the new variant is compared.

What does a 'p-value' represent in A/B testing?

Answer: The probability of seeing results as extreme as observed if there is no real difference between variants

A p-value measures the probability that the observed test results occurred by chance; a low p-value (typically <0.05) suggests statistical significance.

Which is a best practice when running A/B tests?

Answer: Test one element at a time to isolate the variable's effect

Testing one element at a time ensures you can attribute any performance difference directly to that specific change without confounding variables.

What is the 'novelty effect' in A/B testing?

Answer: A temporary spike in engagement for a new variant that fades as users adjust

The novelty effect occurs when users engage more with a new variant simply because it is different, not because it is better, causing inflated early results.