SRE On-Call Practices & Runbooks 4 — Questions and Answers
Question 1: What is a 'canary alert' in the context of on-call alerting strategy?
- An alert that triggers only during business hours
- A synthetic probe alert used to validate that the alerting pipeline itself is functional (Correct answer)
- An alert reserved for critical P0 incidents only
- An alert that automatically pages the CEO
Correct answer: A synthetic probe alert used to validate that the alerting pipeline itself is functional
A canary alert is a deliberately triggered test alert that verifies the end-to-end alerting and paging pipeline is working correctly.
Question 2: Which practice helps ensure runbooks remain accurate over time?
- Locking runbooks after initial publication
- Requiring runbook review as part of every post-mortem and system change (Correct answer)
- Storing runbooks in read-only format
- Assigning one author permanently to each runbook
Correct answer: Requiring runbook review as part of every post-mortem and system change
Linking runbook reviews to post-mortems and change processes ensures documentation is updated when systems or failure patterns change.
Question 3: An alert fires at 3 AM and the on-call engineer determines no user impact exists. What is the appropriate action?
- Immediately page the entire team to investigate
- Silence the alert permanently without investigation
- Acknowledge, document the lack of impact, and create a ticket to tune the alert (Correct answer)
- Escalate to the VP of Engineering
Correct answer: Acknowledge, document the lack of impact, and create a ticket to tune the alert
Non-impacting alerts should be acknowledged and documented, then a follow-up ticket created to reduce noise — silencing without investigation risks missing real issues.
Question 4: What is 'toil' in SRE terminology as it relates to on-call work?
- Any work that improves system reliability
- Repetitive, manual, and automatable operational work that scales with service growth (Correct answer)
- Writing documentation and runbooks
- Performing capacity planning exercises
Correct answer: Repetitive, manual, and automatable operational work that scales with service growth
Toil is repetitive manual work tied to running a production service that provides no enduring value and grows with the system — a primary target for automation.
Question 5: Which on-call handoff practice is most effective for continuity?
- The outgoing engineer deletes their notes to keep documentation clean
- Verbal handoff only, since written notes become stale
- A structured handoff document covering open incidents, recent changes, and known risks (Correct answer)
- Automatic rotation with no explicit handoff process
Correct answer: A structured handoff document covering open incidents, recent changes, and known risks
A structured handoff document ensures the incoming on-call engineer is aware of ongoing issues, recent changes, and system risks without having to rediscover them.
Question 6: In PagerDuty or OpsGenie, what is an 'override' typically used for?
- Permanently reassigning on-call ownership
- Temporarily assigning on-call duty to a different engineer for a defined time window (Correct answer)
- Disabling all alerts for a service
- Changing the escalation policy permanently
Correct answer: Temporarily assigning on-call duty to a different engineer for a defined time window
An override temporarily substitutes one engineer for another in the schedule, used for vacations, sick days, or planned absences without changing the permanent rotation.
Question 7: What is the purpose of a 'blameless post-mortem' in on-call culture?
- To identify which engineer caused the incident so they can be disciplined
- To analyze what went wrong and improve systems without attributing personal fault (Correct answer)
- To calculate the financial cost of the incident
- To file a formal complaint against a vendor
Correct answer: To analyze what went wrong and improve systems without attributing personal fault
Blameless post-mortems focus on systemic causes and process improvements rather than individual blame, fostering a culture where engineers report problems honestly.
SRE On-Call Practices & Runbooks 4
What is a 'canary alert' in the context of on-call alerting strategy?