Conversion Optimization A/B Testing and Experimentation 3

Question 1

What is a 'sequential test' (also called a 'before/after test') and what is its key weakness?

Accepted Answer

Comparing metrics before and after a change, with the weakness being inability to control for external variables

Answer

Sequential tests cannot isolate the effect of the change from external factors like seasonality or marketing campaigns that differ between periods.

Question 2

What is 'novelty effect' in A/B testing?

Accepted Answer

A temporary boost in engagement for a new variant simply because it is new, not because it is better

Answer

Novelty effect causes returning users to interact more with a new variant initially, which fades over time and can produce misleading early results.

Question 3

In Bayesian A/B testing, what does 'probability to be best' represent?

Accepted Answer

The likelihood that a variant is the true winner based on observed data

Answer

In Bayesian testing, probability to be best quantifies our confidence that a variant outperforms others given the collected data.

Question 4

What is 'segment analysis' in A/B testing?

Accepted Answer

Breaking down test results by user groups (device, location, new vs. returning) to find where the effect is strongest

Answer

Segment analysis reveals whether a variant wins broadly or only for specific user groups, enabling more targeted optimization decisions.

Question 5

What is 'interaction effect' when running multiple simultaneous A/B tests?

Accepted Answer

When two tests running at the same time influence each other's results, corrupting both outcomes

Answer

Interaction effects occur when the same users encounter multiple experiments simultaneously, causing the tests to confound each other.

Conversion Optimization Practice Test