Conversion Optimization Advanced Website Conversion Optimization 3 — Questions and Answers
Question 1: Which technique is used to understand WHY users drop off a funnel step rather than just measuring that they do?
- Cohort analysis
- Session recording and user interviews (Correct answer)
- Multivariate testing
- Traffic segmentation
Correct answer: Session recording and user interviews
Quantitative data identifies where drop-offs occur; qualitative methods like session recordings and interviews reveal the underlying reasons.
Question 2: A lead generation form is tested with 3 fields vs. 7 fields. The 3-field variant produces 40% more submissions but leads are 30% less qualified. What should the CRO team do?
- Ship the 3-field form since it wins on the primary metric
- Measure cost-per-qualified-lead to determine the true business winner (Correct answer)
- Add a CAPTCHA to filter unqualified leads
- Extend the test duration for more data
Correct answer: Measure cost-per-qualified-lead to determine the true business winner
When volume and quality diverge, cost-per-qualified-lead or revenue-per-lead reconciles both into a single business metric.
Question 3: What is 'significance vs. practical significance' in the context of CRO?
- Statistical significance measures sample size; practical significance measures test duration
- A result can be statistically significant but too small an effect to justify implementation costs (Correct answer)
- Practical significance applies only to e-commerce, not lead gen
- They are interchangeable terms in most testing platforms
Correct answer: A result can be statistically significant but too small an effect to justify implementation costs
A statistically significant 0.01% lift may cost more to implement than it earns, making it practically insignificant.
Question 4: A/B test results show a p-value of 0.03 for variant B after 14 days. The MDE (minimum detectable effect) was set at 10% but the observed lift is 2%. What is the correct interpretation?
- Variant B is a clear winner and should be shipped
- The test was likely underpowered or the result is a false positive due to peeking (Correct answer)
- A p-value below 0.05 always validates the observed lift
- The MDE should be retroactively adjusted to match the observed lift
Correct answer: The test was likely underpowered or the result is a false positive due to peeking
When observed lift is far below the pre-specified MDE, the test may lack power or be catching noise, not a real effect.
Question 5: Which personalization approach uses real-time behavioral signals (e.g., pages visited, items viewed) to tailor content for anonymous visitors?
- Rule-based segmentation using CRM data
- Behavioral targeting via on-site activity tracking (Correct answer)
- Lookalike audience modeling from paid media data
- Post-purchase email personalization
Correct answer: Behavioral targeting via on-site activity tracking
Behavioral targeting adjusts content dynamically based on what an anonymous visitor does during the current or past sessions without requiring login.
Question 6: An e-commerce site adds social proof (review count and star rating) near the Add-to-Cart button and sees a 9% lift in conversions. Which psychological principle primarily drives this effect?
- Scarcity
- Social proof (conformity heuristic) (Correct answer)
- Reciprocity
- Authority
Correct answer: Social proof (conformity heuristic)
Seeing that many others have purchased and rated a product positively reduces perceived risk through conformity bias.
Question 7: What is the primary risk of running too many simultaneous A/B tests on the same page?
- Increased server load from split traffic
- Interaction effects between tests that make individual results uninterpretable (Correct answer)
- Reduced statistical power for each individual test
- Browser caching conflicts for returning users
Correct answer: Interaction effects between tests that make individual results uninterpretable
Concurrent tests on overlapping page elements can produce interaction effects where the combination of variants—not either variant alone—drives observed changes.
Conversion Optimization Advanced Website Conversion Optimization 3
Which technique is used to understand WHY users drop off a funnel step rather than just measuring that they do?