SRE Toil Reduction & Elimination 5 — Questions and Answers
Question 1: A team inherits a legacy system where every configuration change requires SSH access, manual file edits, and service restarts. Which modernization path BEST eliminates this toil at scale?
- Introduce a configuration management system (e.g., Ansible, Chef, Puppet) with a CI/CD pipeline for config changes (Correct answer)
- Document each manual step more clearly in the runbook
- Assign one engineer as the dedicated 'config admin' to centralize the work
- Automate only the SSH login step using expect scripts
Correct answer: Introduce a configuration management system (e.g., Ansible, Chef, Puppet) with a CI/CD pipeline for config changes
Configuration management tools codify and automate the entire change workflow, replacing error-prone manual steps with auditable, repeatable automation.
Question 2: When an SRE team says 'toil has a 50% cap,' what happens if the team consistently exceeds this cap?
- The team should escalate to management to either increase automation investment or reduce service scope/SLO commitments (Correct answer)
- Engineers should work overtime to complete both toil and project work
- The cap should be raised to match actual workload
- Product teams should be asked to slow feature development
Correct answer: The team should escalate to management to either increase automation investment or reduce service scope/SLO commitments
Exceeding the toil cap is a signal that the team is under-resourced for automation investment or overextended in service ownership — both require management action.
Question 3: Which characteristic of toil differentiates it from a complex incident investigation that also consumes significant engineer time?
- Toil is O(n) repetitive work with no enduring value; incident investigation produces learning, system improvements, and reduced future incidents (Correct answer)
- Incident investigation is always more valuable than any form of toil
- Toil can be automated; incident investigations can never be automated
- There is no difference — both consume engineer time equally
Correct answer: Toil is O(n) repetitive work with no enduring value; incident investigation produces learning, system improvements, and reduced future incidents
Incident investigation produces lasting value through improved reliability, runbooks, and system hardening; toil produces nothing durable when complete.
Question 4: An SRE team builds a chatbot that executes standard remediation playbooks when triggered by engineers via Slack. How does this compare to full automation?
- It reduces toil by eliminating manual CLI steps but retains human judgment for triggering, making it a useful intermediate step toward full automation (Correct answer)
- It increases toil because engineers must now learn the chatbot syntax
- It is equivalent to full automation since the remediation steps are automated
- It violates SRE principles by keeping humans in the loop
Correct answer: It reduces toil by eliminating manual CLI steps but retains human judgment for triggering, making it a useful intermediate step toward full automation
Human-in-the-loop automation is a valid interim stage that reduces execution toil while preserving human oversight for decisions that still require judgment.
Question 5: A company's SRE charter states that 'SREs must spend at least 50% of time on engineering work.' What is the purpose of setting this floor?
- To ensure SREs continuously improve reliability through automation and engineering rather than becoming pure operations staff (Correct answer)
- To limit the number of incidents the team responds to
- To guarantee SREs work no more than 40 hours per week
- To satisfy external compliance requirements for software engineering roles
Correct answer: To ensure SREs continuously improve reliability through automation and engineering rather than becoming pure operations staff
The engineering floor protects against SRE teams devolving into traditional ops teams buried in toil, maintaining their value as reliability engineers.
Question 6: Which of the following BEST describes the relationship between toil reduction and service reliability improvements?
- Reducing toil frees engineering capacity to build more robust systems, better observability, and proactive failure mitigation, directly improving reliability (Correct answer)
- Toil reduction is purely a team morale initiative with no measurable reliability impact
- Reliability improvements always come before toil reduction in priority
- Toil and reliability are independent dimensions that don't influence each other
Correct answer: Reducing toil frees engineering capacity to build more robust systems, better observability, and proactive failure mitigation, directly improving reliability
Toil reduction creates a virtuous cycle: less toil → more engineering time → better reliability work → fewer incidents → less reactive toil.
Question 7: An SRE team successfully automates a toil task that previously took 8 hours/week. Three months later, the automation itself requires 4 hours/week of maintenance. What should the team do?
- Assess whether the 4-hour maintenance burden will decrease as the automation matures, while tracking net savings; refactor or replace the automation if maintenance approaches the original toil cost (Correct answer)
- Accept the result as successful since 4 hours is less than 8 hours
- Revert to manual process since automation introduced its own overhead
- Assign the maintenance to a junior engineer to free senior capacity
Correct answer: Assess whether the 4-hour maintenance burden will decrease as the automation matures, while tracking net savings; refactor or replace the automation if maintenance approaches the original toil cost
Automation that requires high maintenance must be evaluated for long-term net savings; good automation should trend toward near-zero maintenance as it stabilizes.
SRE Toil Reduction & Elimination 5
A team inherits a legacy system where every configuration change requires SSH access, manual file edits, and service restarts.
Which modernization path BEST eliminates this toil at scale?