Conversion Optimization Advanced Website Conversion Optimization 5 — Questions and Answers
Question 1: A retail site tests removing product recommendations from the cart page and sees a 5% lift in checkout completion but a 12% drop in average order value. What is the net business implication?
- Remove recommendations since checkout completion is the primary goal
- Calculate revenue per visitor for each variant to determine which variant is truly better (Correct answer)
- Add more recommendations to recover the AOV loss
- The test is inconclusive and should be rerun
Correct answer: Calculate revenue per visitor for each variant to determine which variant is truly better
Revenue per visitor (or revenue per session) combines conversion rate and order value into a single metric that reflects the true business outcome.
Question 2: Which approach best reduces 'analysis paralysis' caused by too many product options on a category page?
- Adding more detailed filters to give users full control
- Reducing visible options and using progressive disclosure to reveal more on demand (Correct answer)
- Displaying all products with user reviews sorted by rating
- Increasing page load speed to keep users from leaving
Correct answer: Reducing visible options and using progressive disclosure to reveal more on demand
Progressive disclosure presents a manageable choice set first, reducing cognitive overload while keeping the full catalog accessible for power users.
Question 3: What is the purpose of a 'holdout group' in a long-running CRO program?
- A segment excluded from all experiments to measure baseline drift over time (Correct answer)
- A control group for a single A/B test held constant during the run
- A group of high-value users protected from disruptive experiments
- A quality assurance team that reviews test setups before launch
Correct answer: A segment excluded from all experiments to measure baseline drift over time
A holdout group permanently excluded from all tests lets teams measure true cumulative lift from the optimization program versus organic trends.
Question 4: A fintech company's registration flow converts at 8%. A usability study shows users are confused by the 'proof of income' step. Which CRO response is most appropriate?
- Remove the step entirely to maximize conversion volume
- Reorder, rename, and add contextual help text to the step before testing removal (Correct answer)
- Add a progress bar to reassure users
- Delay the step to post-registration with a follow-up email
Correct answer: Reorder, rename, and add contextual help text to the step before testing removal
Clarifying confusing steps through copy, sequencing, and help text should be tested before removing required steps that may be legally or operationally necessary.
Question 5: What is 'velocity' in the context of a CRO testing roadmap?
- The page load speed threshold required before testing can begin
- The number of validated experiments shipped per month, used to forecast optimization impact over time (Correct answer)
- The rate at which A/B test traffic accumulates to reach significance
- The speed at which variant changes are deployed to production
Correct answer: The number of validated experiments shipped per month, used to forecast optimization impact over time
Testing velocity is the throughput metric for a CRO program — more validated tests shipped per period compounds learning and revenue gains over time.
Question 6: Which user research method is most effective for identifying friction in a multi-step onboarding flow for a complex B2B product?
- Click-tracking heatmaps on the first onboarding screen
- Moderated usability testing with think-aloud protocol across the full flow (Correct answer)
- 5-second tests on landing page first impressions
- Exit surveys triggered when users leave the site
Correct answer: Moderated usability testing with think-aloud protocol across the full flow
Moderated usability testing with think-aloud uncovers reasoning, confusion, and decision points at each step of a complex flow that passive tools cannot capture.
Question 7: A company observes that users who see a personalized homepage convert at 6.8% while users who see the generic homepage convert at 4.1%. Why should this NOT be treated as a valid A/B test result?
- The sample sizes may be unequal between the two groups
- Personalized users are self-selected (likely returning or logged-in), creating selection bias (Correct answer)
- The conversion rate difference is too large to be statistically plausible
- Personalization violates GDPR and the data cannot be used
Correct answer: Personalized users are self-selected (likely returning or logged-in), creating selection bias
When users are not randomly assigned to conditions, pre-existing differences (engagement level, intent) drive the observed gap rather than the personalization itself.
Conversion Optimization Advanced Website Conversion Optimization 5
A retail site tests removing product recommendations from the cart page and sees a 5% lift in checkout completion but a 12% drop in average order value.
What is the net business implication?