Product Management Product Metrics and Analytics 3 — Questions and Answers
Question 1: What is 'A/B testing' used for in product management?
- Comparing two versions of a feature to determine which performs better (Correct answer)
- Testing two different products against each other
- Alternating sprint cycles
- Splitting the team into two groups for parallel development
Correct answer: Comparing two versions of a feature to determine which performs better
A/B testing exposes different user segments to two variants of a feature or design and uses statistical analysis to determine which version achieves the desired outcome better.
Question 2: What is 'statistical significance' in the context of product experiments?
- The probability that observed results are due to the change and not random chance (Correct answer)
- Whether a metric exceeded its monthly target
- The size of the user sample tested
- Whether an executive approved the result
Correct answer: The probability that observed results are due to the change and not random chance
Statistical significance (typically p < 0.05) tells a PM that the difference between experiment variants is unlikely to be due to random variation.
Question 3: What is a 'funnel' in product analytics?
- A sequence of steps users take toward a conversion goal, tracked to find drop-off points (Correct answer)
- A method for filtering bug reports
- A visual representation of team structure
- A type of customer survey
Correct answer: A sequence of steps users take toward a conversion goal, tracked to find drop-off points
A product funnel maps the steps from awareness to conversion, allowing PMs to identify which stages lose the most users and prioritize optimization.
Question 4: What does 'Monthly Recurring Revenue' (MRR) track?
- Predictable subscription revenue normalized to a monthly amount (Correct answer)
- One-time purchase revenue per month
- Total revenue across all products
- Marketing spend per month
Correct answer: Predictable subscription revenue normalized to a monthly amount
MRR is the cornerstone SaaS revenue metric representing the predictable, recurring portion of revenue earned each month from active subscriptions.
Question 5: What is 'retention rate' and how does it differ from churn rate?
- Retention rate measures users who stay; churn rate measures users who leave — they are inverses (Correct answer)
- They measure the same thing with different formulas
- Retention rate is quarterly; churn is annual
- Retention tracks revenue; churn tracks users only
Correct answer: Retention rate measures users who stay; churn rate measures users who leave — they are inverses
Retention rate and churn rate are complementary metrics: if churn is 5%, retention is 95%, providing two lenses on the same user staying behavior.
Question 6: What is the purpose of a 'product health dashboard'?
- To give PMs a real-time view of key metrics so they can quickly identify and respond to issues (Correct answer)
- To monitor server uptime only
- To track team member performance
- To display marketing campaign results exclusively
Correct answer: To give PMs a real-time view of key metrics so they can quickly identify and respond to issues
A product health dashboard consolidates critical KPIs — engagement, retention, revenue, errors — in one view, enabling PMs to monitor product performance and act on anomalies quickly.
Product Management Product Metrics and Analytics 3
What is 'A/B testing' used for in product management?