DOL Metrics and KPIs 2 — Questions and Answers
Question 1: A DevOps team tracks 'Mean Time to Detect' (MTTD). What does this metric primarily measure?
- Time to deploy a fix after detection
- Average time from incident occurrence to alert firing (Correct answer)
- Time to resolve a ticket after assignment
- Average time between successive deployments
Correct answer: Average time from incident occurrence to alert firing
MTTD measures the average elapsed time between when an incident begins and when the team receives an alert or discovers the problem.
Question 2: Which metric best reflects the effectiveness of a team's testing and code review process upstream of production?
- Deployment frequency
- Change failure rate (Correct answer)
- Mean time to recovery
- Lead time for changes
Correct answer: Change failure rate
Change failure rate directly measures the percentage of deployments causing production failures, revealing how well pre-production quality gates work.
Question 3: A leader wants to reduce alert fatigue. Which KPI should they focus on to measure progress?
- Total alerts fired per day
- Alert-to-incident ratio (Correct answer)
- P99 latency
- Deployment pipeline duration
Correct answer: Alert-to-incident ratio
Alert-to-incident ratio measures what fraction of alerts correspond to real incidents, so improving it means fewer false positives and less fatigue.
Question 4: What is the primary risk of optimizing a single metric like deployment frequency in isolation?
- Deployments become too infrequent
- Teams may sacrifice quality or stability to boost the number (Correct answer)
- Infrastructure costs increase linearly
- Lead time automatically increases
Correct answer: Teams may sacrifice quality or stability to boost the number
Goodhart's Law warns that when a measure becomes a target, teams game it — shipping trivial changes just to inflate deployment count while ignoring quality.
Question 5: Which approach best ensures SLO (Service Level Objective) thresholds remain meaningful over time?
- Set them once during initial architecture and never change them
- Review and adjust SLOs each quarter based on user feedback and error budget consumption (Correct answer)
- Always set SLOs at 99.999% to signal high standards
- Derive SLOs from infrastructure uptime rather than user experience
Correct answer: Review and adjust SLOs each quarter based on user feedback and error budget consumption
SLOs should be living targets reviewed regularly against actual error budget burn and evolving user expectations to stay actionable.
Question 6: A team's error budget is 80% consumed with two weeks left in the month. What is the recommended DevOps response?
- Ignore it and continue normal release cadence
- Halt risky releases and focus on reliability improvements (Correct answer)
- Increase deployment frequency to flush out bugs faster
- Raise the SLO target to give more budget headroom
Correct answer: Halt risky releases and focus on reliability improvements
When the error budget is nearly exhausted, SRE practice recommends freezing risky deployments until reliability work replenishes the budget.
Question 7: Which metric is most useful for evaluating the efficiency of a CI/CD pipeline itself?
- MTTR
- Pipeline duration and queue wait time (Correct answer)
- Number of open pull requests
- Customer satisfaction score
Correct answer: Pipeline duration and queue wait time
Pipeline duration and queue wait time reveal how long engineers wait for feedback, directly impacting developer productivity and flow.
DOL Metrics and KPIs 2
A DevOps team tracks 'Mean Time to Detect' (MTTD).
What does this metric primarily measure?