Conversion Optimization A/B Testing and Experimentation 3

Question 1

What is a 'sequential test' (also called a 'before/after test') and what is its key weakness?

Accepted Answer

Comparing metrics before and after a change, with the weakness being inability to control for external variables

Answer

Running variants one at a time in separate windows

Answer

Testing on mobile then desktop sequentially

Answer

Splitting users by their first visit date

Question 2

What is 'novelty effect' in A/B testing?

Accepted Answer

A temporary boost in engagement for a new variant simply because it is new, not because it is better

Answer

The first-mover advantage in a competitive market

Answer

The impact of new product launches on test results

Answer

Increased conversions from newly acquired users

Question 3

In Bayesian A/B testing, what does 'probability to be best' represent?

Accepted Answer

The likelihood that a variant is the true winner based on observed data

Answer

The percentage of users assigned to the winning variant

Answer

The probability that the test reached significance

Answer

The chance the test was set up correctly

Question 4

What is 'segment analysis' in A/B testing?

Accepted Answer

Breaking down test results by user groups (device, location, new vs. returning) to find where the effect is strongest

Answer

Dividing the test into multiple time segments

Answer

Splitting the test budget across different channels

Answer

Grouping test pages by URL structure

Question 5

What is 'interaction effect' when running multiple simultaneous A/B tests?

Accepted Answer

When two tests running at the same time influence each other's results, corrupting both outcomes

Answer

When test participants interact with live chat during a test

Answer

When A/B test data integrates with CRM records

Answer

When multiple team members edit the same variant

Question 6

Which tool feature helps prevent 'flicker effect' in client-side A/B testing?

Accepted Answer

Anti-flicker snippet or async hiding technique that hides the page until the variant loads

Answer

Server-side caching of variant HTML

Answer

Using only CSS changes in tests

Answer

Limiting tests to logged-in users only