CEC Analytics & Performance Optimization 3 — Questions and Answers
Question 1: An ecommerce site runs an A/B test on its product page CTA button color. After 2 weeks, the result shows p=0.08. What should the consultant recommend?
- Implement the variant since it's close to significant
- End the test and declare a winner based on revenue alone
- Continue the test to gather more data or accept the null hypothesis (Correct answer)
- Run a follow-up test immediately with a new color
Correct answer: Continue the test to gather more data or accept the null hypothesis
A p-value of 0.08 does not meet the standard 0.05 significance threshold, meaning results are inconclusive and the test should continue or be treated as no significant difference found.
Question 2: Which ecommerce KPI directly measures the percentage of customers who make a second purchase?
- Customer Lifetime Value (CLV)
- Repeat Purchase Rate (Correct answer)
- Net Promoter Score (NPS)
- Customer Acquisition Cost (CAC)
Correct answer: Repeat Purchase Rate
Repeat Purchase Rate calculates the proportion of customers who have placed more than one order, directly measuring loyalty and retention.
Question 3: What is the primary purpose of setting up 'custom dimensions' in a web analytics platform?
- To create custom dashboard layouts
- To capture business-specific data not tracked by default, such as logged-in status or product category (Correct answer)
- To filter out internal traffic automatically
- To connect the platform to a CRM
Correct answer: To capture business-specific data not tracked by default, such as logged-in status or product category
Custom dimensions allow you to send and analyze data unique to your business that the analytics platform doesn't collect out of the box.
Question 4: A store's cart abandonment rate jumps from 65% to 80% after a checkout redesign. What is the most critical next step?
- Increase retargeting ad budget immediately
- Use session recordings and funnel reports to identify where users drop off in the new checkout (Correct answer)
- Revert to the old design without analysis
- Lower product prices to compensate
Correct answer: Use session recordings and funnel reports to identify where users drop off in the new checkout
Session recordings and funnel analysis pinpoint the exact step causing abandonment, enabling targeted fixes rather than guesswork.
Question 5: In ecommerce, 'CLV:CAC ratio' of 3:1 is generally considered:
- Too low, indicating the business is unprofitable
- A healthy benchmark meaning customers generate 3x their acquisition cost (Correct answer)
- A sign that marketing spend should be cut
- Only relevant for subscription businesses
Correct answer: A healthy benchmark meaning customers generate 3x their acquisition cost
A 3:1 CLV to CAC ratio is widely cited as the minimum healthy threshold, meaning each customer generates three times what it cost to acquire them.
Question 6: What does 'incrementality testing' measure in ecommerce marketing analytics?
- The increase in page speed after optimization
- The true lift in conversions caused by a marketing channel versus what would have happened without it (Correct answer)
- The incremental revenue per additional product SKU
- The year-over-year growth in organic traffic
Correct answer: The true lift in conversions caused by a marketing channel versus what would have happened without it
Incrementality testing isolates a channel's true causal impact by comparing a test group exposed to the channel against a holdout group that wasn't.
Question 7: Which tool is best suited for identifying specific user interaction problems on an ecommerce site through visual behavior data?
- Google Search Console
- Heatmap and session recording tools like Hotjar or Microsoft Clarity (Correct answer)
- A/B testing platforms like Optimizely
- Email marketing dashboards
Correct answer: Heatmap and session recording tools like Hotjar or Microsoft Clarity
Heatmap and session recording tools visualize where users click, scroll, and rage-click, revealing UX friction invisible in numeric analytics alone.
CEC Analytics & Performance Optimization 3
An ecommerce site runs an A/B test on its product page CTA button color.
After 2 weeks, the result shows p=0.08.
What should the consultant recommend?