SRE On-Call Escalation and Runbook Design 2 — Questions and Answers
Question 1: What is the 'follow-the-sun' on-call model, and what are its primary benefits and challenges?
- Follow-the-sun distributes on-call coverage across geographically distributed teams in different time zones so that each team handles incidents during their business hours, reducing after-hours pages; the main challenge is handoff quality and context continuity (Correct answer)
- Follow-the-sun is a monitoring strategy that increases alert sensitivity during daylight hours when more users are active
- Follow-the-sun is a cloud region routing policy that routes users to the geographically nearest data center during business hours
- Follow-the-sun refers to the practice of scheduling all deployments during the on-call engineer's daytime hours to ensure maximum availability during risky changes
Correct answer: Follow-the-sun distributes on-call coverage across geographically distributed teams in different time zones so that each team handles incidents during their business hours, reducing after-hours pages; the main challenge is handoff quality and context continuity
Follow-the-sun coverage ensures on-call engineers are never paged during their night by handing off responsibility as business hours shift around the globe — but requires excellent incident handoff processes to prevent information loss during transitions.
Follow-the-sun (FTS) on-call requires teams distributed across at least 3 time zones (e.g., US West, Europe, APAC) so that each team's 8-12 hour on-call window falls entirely within their working hours. Benefits: (1) No on-call engineer works through their night — significantly reduces on-call fatigue and burnout. (2) Faster response during business hours (engineers are already awake and alert). (3) Makes global teams valuable for their time zone coverage, not just their skills. Challenges: (1) Handoff quality: the outgoing team must communicate all open incidents, ongoing investigations, and recent unusual behavior to the incoming team. Poor handoff leads to repeated work or missed context. (2) Knowledge distribution: each regional team must have sufficient expertise for the services they cover — can't have only one global expert. (3) Cross-timezone collaboration during major incidents: complex incidents may require multiple regional teams to collaborate across time zones. (4) Higher team staffing requirements: FTS requires 3× the on-call headcount compared to a global rotation. Handoff tools: standardized handoff templates, incident status pages, shared runbooks, and brief synchronous handoff calls at shift boundaries.
Question 2: What is 'alert routing' in incident management, and why is it important to route alerts to the RIGHT team?
- Alert routing directs each alert to the team with the knowledge and access to resolve it, reducing time-to-acknowledge and preventing incidents from sitting in an un-monitored queue while being routed to the wrong team (Correct answer)
- Alert routing distributes alerts evenly across all on-call engineers to ensure equal workload distribution
- Alert routing is the process of converting monitoring alerts into ticketing system issues for SLA tracking
- Alert routing filters alerts to only notify engineers when the alert has persisted for more than 5 minutes to reduce false positives
Correct answer: Alert routing directs each alert to the team with the knowledge and access to resolve it, reducing time-to-acknowledge and preventing incidents from sitting in an un-monitored queue while being routed to the wrong team
Routing alerts to the wrong team wastes critical response time — the team that receives the alert may spend time just figuring out who owns the service, delaying diagnosis and remediation.
Alert routing is the configuration that determines which on-call schedule receives which alert. Poor routing creates: (1) Routing delay: wrong team receives alert, then must route it to the right team — adding 5-15 minutes to MTTD. (2) Knowledge gaps: the receiving team may not have the expertise or access to resolve the issue. (3) Frustration and alert fatigue: teams that frequently receive alerts for services they don't own start ignoring alerts, causing real incidents to be missed. (4) SLA violations: routing delays can push resolution times past SLA thresholds. Alert routing best practices: (1) Service catalog ownership: every service has a defined owner team. (2) Automated routing rules: monitoring tools (PagerDuty, Opsgenie) route each alert to the team's on-call schedule based on service tags. (3) Escalation paths defined per service: if primary team doesn't respond, route to secondary team or incident commander. (4) Regular audit: review routing rules when teams are reorganized or services change ownership. (5) 'You build it, you run it' principle: the team that builds a service owns its on-call alerts.
Question 3: A runbook step says: 'Check if the issue is in the database or application layer.' What makes this a POOR runbook instruction?
- It is vague and non-actionable — a good runbook instruction would specify exactly which commands to run, which metrics to check, and how to interpret the results to make the determination (Correct answer)
- It asks the on-call engineer to make a judgment call, which is appropriate in runbooks for experienced engineers
- It is too simple — runbook instructions should always cover all possible diagnostic paths
- It conflates two separate diagnostic procedures that should be in separate runbooks
Correct answer: It is vague and non-actionable — a good runbook instruction would specify exactly which commands to run, which metrics to check, and how to interpret the results to make the determination
Vague instructions require domain knowledge that a stressed, sleep-deprived on-call engineer may not have at 3 AM. A good runbook specifies exact commands, expected outputs, and decision criteria so the procedure can be executed mechanically.
A runbook instruction should be specific enough that it is mechanically executable, not a task that requires prior knowledge to interpret. 'Check if the issue is in the database or application layer' fails because it doesn't specify: What to check: which metrics, logs, or dashboards? What commands to run: 'check database latency' — which query, which tool (psql, AWS Console, Datadog)? What to look for: what constitutes 'database layer' vs. 'application layer' in this specific service? What to do with the result: if it IS the database, what next? If it IS the application, what next? Better runbook instruction: 'Run: kubectl logs deployment/my-app --since=5m | grep ERROR. If log shows "connection refused to postgres-primary", proceed to Step 4 (database diagnostic). If log shows "NullPointerException" or other application error, proceed to Step 6 (application diagnostic). If no errors in logs, proceed to Step 7 (external dependency check).' This gives exact commands, expected output patterns, and unambiguous decision branches.
Question 4: What is 'context-sensitive escalation' and how does it improve on simple time-based escalation?
- Context-sensitive escalation considers the severity, service type, and current conditions (e.g., high-traffic event, recent deployment) to escalate differently depending on context — a high-traffic event may trigger faster escalation than a normal business day (Correct answer)
- Context-sensitive escalation uses the on-call engineer's location (context) to determine who should be paged next, always routing to the nearest team member
- Context-sensitive escalation requires the on-call engineer to explain the incident context before escalation is approved, ensuring only valid escalations are made
- Context-sensitive escalation delays all escalations by a configurable amount based on the time of day, reducing middle-of-the-night pages
Correct answer: Context-sensitive escalation considers the severity, service type, and current conditions (e.g., high-traffic event, recent deployment) to escalate differently depending on context — a high-traffic event may trigger faster escalation than a normal business day
Simple time-based escalation ('escalate if unresolved after 30 minutes') is inflexible. Context-sensitive escalation adjusts escalation urgency based on incident severity, business context (peak traffic periods), recent changes, and affected user tier.
Context-sensitive escalation policies consider multiple factors: (1) Severity: P1 incidents escalate faster (15 min) than P3 incidents (next business day). (2) Business context: during defined peak periods (product launches, Black Friday, earnings release), all escalation timers are shortened because the cost of extended downtime is higher. (3) Recent deployment: if a deployment occurred in the last 30 minutes and an incident fires, escalate faster (the deploying team should be immediately involved). (4) Affected user tier: incidents affecting enterprise customers may escalate faster than incidents affecting free-tier users, based on SLA commitments. (5) Blast radius: incidents affecting payment processing escalate differently than incidents affecting internal reporting tools. Implementation: PagerDuty and Opsgenie support conditional escalation policies. More sophisticated implementations use incident management platforms that can evaluate real-time context signals (current traffic level, recent change log, on-call team's location) to adjust escalation timing dynamically.
Question 5: What should a runbook include for an 'escalation decision tree' to be effective?
- Clear decision points based on observable conditions (e.g., 'if mitigation not found in 20 min OR user impact >10%'), explicit escalation targets by name/role, contact information, and what information to share when escalating (Correct answer)
- A list of all possible failure scenarios and the corresponding fix for each, to avoid the need for escalation entirely
- A ranking of all engineers on the team by their expertise level for the service
- Approval workflows for escalation decisions to prevent unnecessary interruptions to senior engineers
Correct answer: Clear decision points based on observable conditions (e.g., 'if mitigation not found in 20 min OR user impact >10%'), explicit escalation targets by name/role, contact information, and what information to share when escalating
An effective escalation decision tree removes ambiguity: the conditions for escalation are observable facts (not judgment calls), the escalation target is specific (not 'the database team'), and the information to communicate is specified (not assumed).
An effective escalation decision tree in a runbook includes: (1) Observable escalation triggers: not 'if the issue seems serious' but 'if error rate exceeds 5% OR if impact has lasted more than 20 minutes without mitigation.' These are facts, not judgments. (2) Specific escalation targets: not 'escalate to the database team' but 'page [Database On-Call schedule in PagerDuty] AND notify #incident-response Slack channel.' (3) Contact alternatives: if PagerDuty escalation fails, have an alternate contact method. (4) Information to share: a template for what to tell the escalation target — current symptoms, what has been tried, what is needed from them. This prevents the time-wasting 'what's happening?' conversation. (5) Authority clarification: after escalation, does the escalated engineer take over, or does the original engineer remain the point of contact? (6) Parallel vs. sequential: should multiple escalation paths run simultaneously for P1 incidents (e.g., page database expert AND notify incident commander) or sequentially? Clear answers to these questions mean escalation happens smoothly even when the primary on-call is stressed and sleep-deprived.
Question 6: What is 'incident commander' (IC) role and what are the TOP THREE responsibilities during a major incident?
- Coordinate all responders and communications, make decisions on mitigation actions and escalations, and maintain situational awareness across the incident — NOT to personally diagnose or fix the technical issue (Correct answer)
- Write the postmortem, manage the on-call rotation, and deploy the hotfix that resolves the incident
- Monitor all metrics dashboards, approve all production changes, and write the final root cause analysis
- Contact affected customers, refund SLA credits, and brief the board of directors on the incident impact
Correct answer: Coordinate all responders and communications, make decisions on mitigation actions and escalations, and maintain situational awareness across the incident — NOT to personally diagnose or fix the technical issue
The IC's value is coordination and decision-making authority, not technical execution. By delegating technical tasks, the IC maintains the bird's-eye view needed to make tactical decisions and keep the incident moving toward resolution.
The Incident Commander role, borrowed from the Incident Command System (ICS) used by fire departments and emergency services, has three core functions during a major incident: (1) Coordinate responders: assign tasks clearly ('Alice, you own database investigation; Bob, you own the load balancer'), prevent duplicate work, and ensure every critical track has an owner. (2) Make decisions and maintain authority: decide when to escalate, when to roll back a change, when to declare customer impact, and when to transition from mitigation to root cause analysis — other responders execute, the IC decides. (3) Maintain situational awareness: track what is known, what is being investigated, what has been tried, and what is the current state — ensuring the team doesn't lose track of overall progress while focused on individual tasks. The IC does NOT personally fix technical issues — doing so would pull them out of the coordination role and leave the incident without direction. The IC also manages communications (status page updates, stakeholder notifications) by delegating to a communications lead.
SRE On-Call Escalation and Runbook Design 2
What is the 'follow-the-sun' on-call model, and what are its primary benefits and challenges?