SRE On-Call Escalation and Runbook Design 1 — Questions and Answers
Question 1: What is the 'primary and secondary on-call' model, and what is the role of the secondary on-call engineer?
- The primary on-call responds first to all alerts; the secondary serves as backup if the primary cannot be reached within a defined timeout, and may assist on high-severity incidents requiring multiple responders (Correct answer)
- The secondary on-call monitors all alerts and decides which ones to forward to the primary, acting as a filter
- The secondary on-call handles alerts during business hours while the primary handles after-hours alerts only
- The secondary on-call is responsible only for change management approvals, not incident response
Correct answer: The primary on-call responds first to all alerts; the secondary serves as backup if the primary cannot be reached within a defined timeout, and may assist on high-severity incidents requiring multiple responders
The primary handles the alert first; the secondary is the defined escalation path if the primary is unresponsive or the incident requires more than one responder — this structure is defined in advance so escalation is not ad hoc.
The primary/secondary on-call model provides two layers of coverage: Primary on-call: receives all alerts first, acknowledges within the response SLA (typically 5-15 minutes for P1), begins investigation and remediation. Secondary on-call: escalated to automatically if the primary does not acknowledge within the timeout, or manually by the primary when the incident is too complex for one responder. May also act as Incident Commander for major incidents, freeing the primary to focus on technical investigation. Additional benefits: the secondary reviews the primary's actions during major incidents to catch mistakes under pressure (four-eyes principle); the secondary is available as a human escalation path when the primary needs a second opinion; the secondary may take over if the primary needs to rest during a prolonged incident. Rotation structure: primary and secondary typically rotate independently (e.g., weekly), with the secondary one week behind the primary in the rotation so each person has served as primary before becoming secondary.
Question 2: What information should ALWAYS be included in a runbook's 'prerequisites' section?
- Required access (AWS account, Kubernetes cluster, database), credentials locations, tools to install, and any context needed before the first step (e.g., which monitoring dashboard to check) (Correct answer)
- The author's name, creation date, and last review date for version control purposes
- A glossary of all technical terms used in the runbook for team members unfamiliar with the system
- A list of all services that depend on the service described in the runbook
Correct answer: Required access (AWS account, Kubernetes cluster, database), credentials locations, tools to install, and any context needed before the first step (e.g., which monitoring dashboard to check)
Prerequisites ensure the responder can actually execute the runbook without blocking on missing access or unknown tool requirements — especially critical at 3 AM when getting help to gain access may be impossible.
A runbook without prerequisites fails in practice because responders hit blockers before they can start: Required access: which AWS IAM role, Kubernetes namespace, database user, and monitoring system access are needed? Access that requires a ticket or approval process must be provisioned before the incident happens, not during it. Credentials: where are the credentials stored? (Vault path, AWS Secrets Manager ARN, LastPass folder) — specify the exact location, not just 'find the database password.' Tools: which CLI tools are needed? (kubectl, psql, aws-cli) — are they installed on the on-call team's machines or only on specific jump hosts? Context: which service, which environment (prod/staging), which region? Which monitoring dashboard shows the service's health? Pre-incident preparation: prerequisites also serve as a checklist for new team members joining the on-call rotation — they should verify they have all prerequisite access before their first on-call shift.
Question 3: An on-call engineer receives a P1 alert at 2 AM but cannot determine the cause after 20 minutes of investigation. What is the CORRECT action?
- Escalate to the secondary on-call or the designated expert as defined in the escalation policy, without waiting longer — unresolved P1 incidents require additional resources (Correct answer)
- Continue investigating independently for another hour before escalating to avoid waking others unnecessarily
- Resolve the alert as 'no issue found' and create a low-priority ticket for investigation during business hours
- Attempt to mitigate by restarting all services in the affected stack, then monitor to see if the alert clears
Correct answer: Escalate to the secondary on-call or the designated expert as defined in the escalation policy, without waiting longer — unresolved P1 incidents require additional resources
P1 incidents have defined escalation timeouts in the incident management policy — waiting beyond them is a policy violation that risks extended user impact. Escalation is not failure; it is the defined process.
The escalation policy defines escalation triggers and timelines for exactly this situation. For a P1 incident (severe user impact), typical policy is: 0 min: alert fires, primary on-call acknowledges. 15-30 min: if no mitigation path identified, escalate to secondary or subject matter expert. 30-60 min: if not mitigated, Incident Commander declared for major incident, communications lead activated. Waiting beyond the defined escalation window: wastes time that could be spent with additional expertise, may allow user impact to compound, and violates the incident management SLA. Dismissing the alert (option C) is dangerous for a P1 — if the issue is real, it will worsen. Randomly restarting services (option D) without understanding the cause destroys forensic evidence, may worsen the incident, and creates false confidence if the alert clears temporarily. The correct action acknowledges the individual's limitations and leverages the team's collective expertise through defined escalation.
Question 4: What is the 'severity matrix' in incident management, and why must it be defined BEFORE incidents occur?
- A severity matrix defines criteria for each severity level (P1-P4) based on user impact, service criticality, and business risk — predefined so engineers make consistent, non-emotional severity decisions during high-pressure incidents (Correct answer)
- A severity matrix is a post-incident tool that retrospectively rates the severity of past incidents for trend analysis
- A severity matrix is a tool for estimating the engineering effort required to fix a known issue
- A severity matrix defines how many on-call engineers should be on the rotation based on team size
Correct answer: A severity matrix defines criteria for each severity level (P1-P4) based on user impact, service criticality, and business risk — predefined so engineers make consistent, non-emotional severity decisions during high-pressure incidents
Pre-defined severity criteria ensure consistent escalation and response across all incidents and all engineers — without it, severity is decided emotionally or inconsistently, leading to under-escalating serious incidents or over-escalating minor ones.
A severity matrix defines objective criteria for each severity level. Example: P1 (Critical): >20% of users affected, core business function unavailable, revenue directly impacted → page primary + secondary + leadership, declare major incident. P2 (High): 5-20% of users affected OR core function degraded → page primary, escalate after 30 min if unresolved. P3 (Medium): <5% of users affected OR non-core function impacted → create ticket, fix within 24 hours. P4 (Low): no user impact, potential future risk → fix in next sprint. Benefits of pre-definition: (1) Consistency: all engineers assess severity the same way. (2) Appropriate escalation: P1 criteria trigger the right escalation path immediately. (3) Removes pressure to 'downgrade' severity to avoid waking people up — criteria are objective. (4) Enables SLA tracking: each severity level has defined response times. (5) Provides data for trend analysis. The matrix should be reviewed and updated as the service and business impact understanding evolves.
Question 5: What is 'on-call shadowing' and why is it a recommended practice for new team members?
- On-call shadowing pairs new team members with experienced on-call engineers, allowing them to observe incident response and use runbooks in real situations before taking primary on-call responsibility (Correct answer)
- On-call shadowing is a security audit practice where senior engineers secretly monitor on-call engineers' actions
- On-call shadowing requires new team members to work back-to-back on-call shifts to rapidly build experience
- On-call shadowing means the on-call engineer should always have a senior engineer available on video call during their shift
Correct answer: On-call shadowing pairs new team members with experienced on-call engineers, allowing them to observe incident response and use runbooks in real situations before taking primary on-call responsibility
Shadowing provides safe, low-stakes learning — new team members see real incidents handled, understand how runbooks are used in practice, and build confidence before they hold primary on-call responsibility.
On-call shadowing is a mentorship practice for onboarding new team members to the on-call rotation: (1) New team member joins the on-call rotation as 'shadow' (no responsibility) for 2-4 weeks. (2) They observe the primary on-call respond to all alerts — watching how they triage, diagnose, use runbooks, and escalate. (3) They participate in post-incident reviews and postmortem discussions. (4) Near the end of the shadow period, they may handle some alerts with the experienced engineer watching (reverse shadow). (5) Then they take primary on-call with the experienced engineer as backup for the first real rotation. Benefits: (1) Safety: the primary handles incidents while the new member learns — no risk of a new engineer making an inexperienced mistake alone on P1. (2) Realistic learning: actual incidents are more educational than simulations. (3) Runbook validation: new members often spot runbook gaps ('this step doesn't work anymore') because they follow them more literally. (4) Psychological safety: the new member builds confidence before being responsible alone.
Question 6: Which metric is MOST useful for identifying that the on-call rotation is unsustainable for the team?
- Number of pages per on-call shift exceeding the team's policy limit (e.g., more than 5 pages per night shift), indicating that on-call load is too high to allow adequate rest (Correct answer)
- The percentage of incidents that require escalation to secondary on-call
- The average time from alert to acknowledgment across all incidents
- The ratio of false positive alerts to true positive alerts
Correct answer: Number of pages per on-call shift exceeding the team's policy limit (e.g., more than 5 pages per night shift), indicating that on-call load is too high to allow adequate rest
Pages-per-shift that consistently exceed the sustainable limit (typically 2-3 actionable pages per night per Google's guidelines) directly indicate alert volume that prevents adequate sleep and recovery, making the rotation unsustainable.
Google's SRE book sets a guideline of no more than 2-3 significant pages per 12-hour on-call shift on average (with never more than 5 per shift). Exceeding this regularly indicates the rotation is unsustainable because: (1) Engineers cannot get adequate sleep during night shifts if paged multiple times per night. (2) Chronic sleep deprivation leads to errors, burnout, and attrition. (3) Fatigue during incident response increases the risk of mistakes that worsen incidents. Metrics to track: number of pages per shift (especially night shifts), time engaged per shift (if an engineer is actively working for more than 25% of an on-call shift, that's too much), and time to sleep recovery after an on-call shift. When pages per shift consistently exceed the sustainable limit, the team should: reduce alert noise (false positives and non-actionable alerts), automate common responses (reduce time-to-resolve), add runbook coverage (reduce time per incident), and potentially grow the team or reduce service scope. The escalation rate metric (option B) measures collaboration effectiveness; MTTD (option C) measures monitoring speed; false positive rate (option D) measures alert quality — all useful but less directly tied to sustainability.
SRE On-Call Escalation and Runbook Design 1
What is the 'primary and secondary on-call' model, and what is the role of the secondary on-call engineer?