CWA CWA Conversion Rate Optimization & A/B Testing 2 — Questions and Answers
Question 1: What is a 'hypothesis' in A/B testing?
- A proven fact about user behavior
- A testable prediction about how a change will affect user behavior (Correct answer)
- A statistical formula for measuring conversions
- A tool for splitting test traffic
Correct answer: A testable prediction about how a change will affect user behavior
An A/B test hypothesis is a structured, testable prediction stating that a specific change will produce a measurable improvement in a defined metric.
Question 2: What is 'multivariate testing' (MVT) in CRO?
- Testing multiple websites simultaneously
- Testing multiple changes across multiple page elements at the same time (Correct answer)
- Testing with multiple user demographic segments
- Running multiple sequential A/B tests
Correct answer: Testing multiple changes across multiple page elements at the same time
Multivariate testing simultaneously tests multiple variations of multiple page elements to identify which combination produces the best results.
Question 3: What is the 'control' in an A/B test?
- The new variant being evaluated
- The original, unchanged version of the page (Correct answer)
- The traffic allocation percentage
- The testing platform being used
Correct answer: The original, unchanged version of the page
The control is the existing, unmodified version of the page that serves as the baseline against which the new variant is compared.
Question 4: What does a 'p-value' represent in A/B testing?
- The percentage of visitors who converted
- The probability of seeing results as extreme as observed if there is no real difference between variants (Correct answer)
- The price per acquired customer
- The page value metric in analytics
Correct answer: The probability of seeing results as extreme as observed if there is no real difference between variants
A p-value measures the probability that the observed test results occurred by chance; a low p-value (typically <0.05) suggests statistical significance.
Question 5: Which is a best practice when running A/B tests?
- Test multiple page elements simultaneously in a single test
- Stop the test as soon as a positive result appears
- Test one element at a time to isolate the variable's effect (Correct answer)
- Always limit test duration to exactly 24 hours
Correct answer: Test one element at a time to isolate the variable's effect
Testing one element at a time ensures you can attribute any performance difference directly to that specific change without confounding variables.
Question 6: What is the 'novelty effect' in A/B testing?
- When users prefer a new feature because it is genuinely superior
- A temporary spike in engagement for a new variant that fades as users adjust (Correct answer)
- The effect of testing too many variants simultaneously
- A statistical adjustment for small sample sizes
Correct answer: A temporary spike in engagement for a new variant that fades as users adjust
The novelty effect occurs when users engage more with a new variant simply because it is different, not because it is better, causing inflated early results.
CWA CWA Conversion Rate Optimization & A/B Testing 2
What is a 'hypothesis' in A/B testing?