Product Management Certification Program Product Metrics & Analytics 2 — Questions and Answers
Question 1: What is the purpose of A/B testing in product management?
- To test whether the product works on both Android and iOS
- To compare two versions of a feature with real users to determine which performs better against a metric (Correct answer)
- To test both the frontend and backend simultaneously
- To compare two sprint team's productivity
Correct answer: To compare two versions of a feature with real users to determine which performs better against a metric
A/B testing (split testing) exposes different user segments to two variants of a feature or design and measures their behavior to determine which version achieves better outcomes.
Question 2: What is 'statistical significance' in the context of A/B testing?
- Having enough test cases in the backlog
- A measure of confidence that an observed difference in results is real and not due to random chance (Correct answer)
- A requirement to run tests for exactly 30 days
- The minimum sample size of 1,000 users
Correct answer: A measure of confidence that an observed difference in results is real and not due to random chance
Statistical significance indicates the probability that the difference between test variants is real, typically requiring p < 0.05 to conclude results are not due to random variation.
Question 3: What is a 'funnel' in product analytics?
- A report on top-of-funnel marketing spend
- A sequence of steps users must complete to achieve a goal, used to measure conversion and drop-off (Correct answer)
- A method for filtering support tickets
- A database query optimization technique
Correct answer: A sequence of steps users must complete to achieve a goal, used to measure conversion and drop-off
A product funnel maps the steps users take toward a desired outcome (such as completing a purchase or onboarding), allowing teams to measure where users progress or drop off.
Question 4: What does 'retention rate' measure for a product?
- How many employees stayed at the company
- The percentage of users who continue using the product after an initial period (Correct answer)
- The rate of data stored per user
- The number of features retained after a redesign
Correct answer: The percentage of users who continue using the product after an initial period
Retention rate measures what proportion of acquired users remain active over a defined time period, indicating whether the product delivers enough ongoing value to keep users coming back.
Question 5: What is 'feature adoption rate'?
- The number of users who request a feature
- The percentage of active users who use a specific feature within a defined time period (Correct answer)
- The number of sprints needed to launch a feature
- The rate at which features are removed from the backlog
Correct answer: The percentage of active users who use a specific feature within a defined time period
Feature adoption rate measures what fraction of the user base is actively using a specific feature, helping PMs evaluate whether features are delivering their intended value.
Question 6: What does 'AARRR' (Pirate Metrics) stand for in product analytics?
- Attract, Acquire, Retain, Refer, Revenue
- Acquisition, Activation, Retention, Referral, Revenue (Correct answer)
- Analyze, Act, Review, Report, Respond
- Awareness, Adoption, Repeat, Recommend, ROI
Correct answer: Acquisition, Activation, Retention, Referral, Revenue
AARRR (Pirate Metrics), coined by Dave McClure, maps the customer lifecycle into five stages: Acquisition, Activation, Retention, Referral, and Revenue.
Product Management Certification Program Product Metrics & Analytics 2
What is the purpose of A/B testing in product management?