SRE Toil Reduction & Elimination 2 — Questions and Answers
Question 1: An SRE team spends 30% of their time manually restarting crashed microservices. Which automation approach BEST eliminates this toil?
- Implement liveness probes and automatic pod restarts in Kubernetes (Correct answer)
- Document the restart procedure in a runbook
- Create a Slack alert for on-call engineers
- Schedule weekly manual health checks
Correct answer: Implement liveness probes and automatic pod restarts in Kubernetes
Liveness probes with automatic restarts fully automate the detection and remediation, eliminating the manual toil entirely.
Question 2: What distinguishes 'overhead' from 'toil' in the SRE framework?
- Overhead has lasting value but no direct production impact, while toil is repetitive manual work scaling with service growth (Correct answer)
- Overhead always requires automation; toil is acceptable at any level
- Toil includes strategic planning; overhead is only operational work
- There is no meaningful distinction between the two terms
Correct answer: Overhead has lasting value but no direct production impact, while toil is repetitive manual work scaling with service growth
Overhead refers to administrative work (meetings, HR tasks) that has organizational value but doesn't scale with service load, unlike toil which grows proportionally.
Question 3: A team's on-call rotation handles 50 alerts per week, 80% of which are auto-resolved before engineers investigate. What should be the FIRST step?
- Silence the alerts that always auto-resolve and add automation to handle them (Correct answer)
- Hire more on-call engineers to share the load
- Increase alert thresholds to reduce noise
- Archive the alerts for monthly review
Correct answer: Silence the alerts that always auto-resolve and add automation to handle them
Alerts that consistently auto-resolve without human action are pure toil — they should be automated end-to-end or suppressed with automated remediation.
Question 4: Which metric BEST measures the impact of a toil-reduction initiative over time?
- Percentage of engineering time spent on toil before and after automation (Correct answer)
- Number of automation scripts written
- Total lines of code added to the automation framework
- Reduction in on-call headcount
Correct answer: Percentage of engineering time spent on toil before and after automation
Tracking the percentage of engineering time consumed by toil directly measures whether the initiative is freeing engineers for higher-value work.
Question 5: A runbook step says 'run this SQL query monthly to archive old records.' How should an SRE classify and address this?
- Classify as toil and automate with a scheduled cron job or database maintenance policy (Correct answer)
- Keep it in the runbook since it only runs once a month
- Delegate it permanently to the database team
- Convert it to a quarterly task to reduce frequency
Correct answer: Classify as toil and automate with a scheduled cron job or database maintenance policy
Repetitive manual tasks with predictable triggers are classic toil regardless of frequency and should be automated away.
Question 6: When evaluating whether to automate a toil task, which factor most strongly justifies NOT automating it immediately?
- The automation effort exceeds the cumulative toil cost over the next two years (Correct answer)
- The task is performed by a junior engineer
- Management hasn't approved the automation project
- The task involves touching a production database
Correct answer: The automation effort exceeds the cumulative toil cost over the next two years
Cost-benefit analysis is essential — if automation takes longer to build than the toil it would save, it may not be worth prioritizing over other investments.
Question 7: An SRE team automates certificate renewal but still manually verifies the renewal succeeded each month. What toil remains?
- The manual verification step is residual toil that should be replaced with automated monitoring and alerting (Correct answer)
- The verification is necessary quality assurance and not toil
- Manual verification is required for compliance and cannot be automated
- The automation itself creates new toil that outweighs the benefit
Correct answer: The manual verification step is residual toil that should be replaced with automated monitoring and alerting
Any repetitive manual check that doesn't require human judgment is toil — automated monitoring with alerting on failure should replace the manual verification.
SRE Toil Reduction & Elimination 2
An SRE team spends 30% of their time manually restarting crashed microservices.
Which automation approach BEST eliminates this toil?