Conversion Optimization Advanced Website Conversion Optimization 2 — Questions and Answers
Question 1: A SaaS company runs an A/B test where variant B increases trial signups by 12% but decreases paid conversions by 8%. What is the most appropriate decision?
- Launch variant B since signups increased
- Pause and analyze downstream revenue impact before deciding (Correct answer)
- Revert to control since paid conversions declined
- Run a follow-up test with a larger sample
Correct answer: Pause and analyze downstream revenue impact before deciding
Top-of-funnel gains that erode revenue metrics require full-funnel analysis before shipping a winner.
Question 2: Which statistical concept explains why stopping an A/B test as soon as significance is reached inflates false positive rates?
- Regression to the mean
- Peeking problem (optional stopping) (Correct answer)
- Simpson's paradox
- Survivorship bias
Correct answer: Peeking problem (optional stopping)
Repeatedly checking p-values and stopping when p<0.05 is hit dramatically increases Type I error beyond the nominal 5%.
Question 3: A checkout page has a 68% cart abandonment rate. Heatmap data shows users pause longest on the shipping cost field. What is the highest-priority hypothesis to test?
- Add a progress bar to the checkout flow
- Display free shipping eligibility or a shipping cost estimator earlier (Correct answer)
- Change the CTA button color to green
- Reduce the number of form fields
Correct answer: Display free shipping eligibility or a shipping cost estimator earlier
Friction identified at a specific field should be addressed directly by removing uncertainty around that cost at that point.
Question 4: What does a multi-armed bandit algorithm offer over classic A/B testing in conversion optimization?
- Eliminates the need for statistical significance thresholds
- Dynamically reallocates traffic to better-performing variants during the test (Correct answer)
- Requires no sample size calculation
- Automatically segments users by persona
Correct answer: Dynamically reallocates traffic to better-performing variants during the test
Multi-armed bandits exploit winners in real time, reducing opportunity cost compared to a fixed 50/50 split.
Question 5: A landing page converts at 3.2% on desktop but 1.1% on mobile. The mobile session share is 65%. Which action delivers the greatest expected revenue lift?
- Optimize desktop layout for the high-converting segment
- Prioritize mobile UX improvements given the large traffic share (Correct answer)
- Create a separate subdomain for mobile users
- Increase desktop ad spend to grow the higher-converting segment
Correct answer: Prioritize mobile UX improvements given the large traffic share
A large traffic segment with a low conversion rate represents the highest absolute opportunity, even if the per-session rate is lower.
Question 6: In CRO, what is a 'novelty effect' and how does it impact test validity?
- When returning users behave differently from new users, skewing results toward familiar designs
- An initial spike in conversions for a new variant driven by user curiosity rather than genuine preference (Correct answer)
- A statistical artifact caused by unequal traffic splits between variants
- The tendency for test winners to underperform in production
Correct answer: An initial spike in conversions for a new variant driven by user curiosity rather than genuine preference
Novelty effects inflate early variant performance; running tests long enough to include returning users reduces this bias.
Question 7: A product detail page redesign lifts add-to-cart rate by 20% but the overall revenue per visitor stays flat. What is the most likely explanation?
- The test sample size was too small
- The redesign attracted lower-intent users who added but did not purchase (Correct answer)
- The payment gateway had latency issues
- Seasonality confounded the results
Correct answer: The redesign attracted lower-intent users who added but did not purchase
A micro-conversion increase that does not translate to revenue often means the funnel step is attracting less-committed users.
Conversion Optimization Advanced Website Conversion Optimization 2
A SaaS company runs an A/B test where variant B increases trial signups by 12% but decreases paid conversions by 8%.
What is the most appropriate decision?