SRE Monitoring & Incident Response 2 — Questions and Answers
Question 1: During a major incident, team members are proposing fixes simultaneously in a group chat, leading to conflicting changes in production. Which incident management structure BEST prevents this?
- Designate a single Incident Commander who coordinates all actions and approves changes during the incident (Correct answer)
- Require a formal change request for every action taken during an incident
- Mute the group chat and have each engineer work independently until the incident resolves
- Escalate all decisions to the VP of Engineering to ensure proper authority
Correct answer: Designate a single Incident Commander who coordinates all actions and approves changes during the incident
An Incident Commander (IC) is the single point of coordination during an incident. The IC approves changes, delegates tasks, and prevents conflicting parallel actions — a core principle from ICS and SRE incident management.
The Incident Commander role, borrowed from the Incident Command System used by emergency services, is the cornerstone of effective incident management. The IC does not necessarily fix the problem — they coordinate the responders, approve changes to prevent conflicting actions, manage communications, and track the state of the incident. Without a clear IC, incidents often devolve into 'too many cooks' scenarios where responders accidentally counteract each other's mitigations. Formal change requests (option B) add too much friction during a fast-moving outage. Independent work (option C) eliminates coordination entirely. Escalating all decisions to senior leadership (option D) creates a bottleneck that slows incident resolution.
Question 2: An alert fires at 3 AM but the on-call engineer determines the issue self-healed before they could investigate. This happens frequently. What is the MOST appropriate action?
- Adjust the alert threshold or add a sustained duration requirement to reduce false positives (Correct answer)
- Delete the alert because it is not actionable
- Lower the alert severity to avoid waking on-call engineers for minor issues
- Add more alerts to capture the root cause of the transient spikes
Correct answer: Adjust the alert threshold or add a sustained duration requirement to reduce false positives
Alerts that consistently fire for self-healing issues represent false positives that contribute to alert fatigue. Adjusting thresholds or requiring the condition to persist before alerting reduces noise while still catching real incidents.
Alert fatigue is one of the most dangerous failure modes in on-call operations — when engineers are repeatedly woken for non-actionable alerts, they begin to ignore or dismiss alerts, increasing the risk of missing real incidents. The correct response to a frequently self-healing alert is to tune it: either raise the threshold so only more severe deviations trigger it, require the condition to persist for a minimum duration (e.g., 5 minutes) before alerting, or change it to a ticket/warning rather than a page. Simply deleting the alert (option B) may remove a useful signal entirely. Lowering severity (option C) doesn't address the root cause of the noise. Adding more alerts (option D) increases toil without addressing the core problem.
Question 3: What is the PRIMARY purpose of a 'communications lead' role during a major incident?
- To handle all stakeholder updates, status page posts, and external communications so the technical responders can focus on remediation (Correct answer)
- To write the postmortem report during the incident while it is still fresh
- To manage the on-call rotation and page additional responders when needed
- To log all actions taken in the incident channel for later review
Correct answer: To handle all stakeholder updates, status page posts, and external communications so the technical responders can focus on remediation
The communications lead shields the technical team from interruptions by managing stakeholder updates, status page postings, and customer communications, allowing engineers to focus on resolving the incident.
During a major incident, business stakeholders, account managers, and executives frequently demand updates, which can severely disrupt engineers who are actively diagnosing and remediating. The communications lead role exists specifically to absorb these interruptions. They craft and send updates to the status page, respond to internal escalations, draft customer-facing communications, and provide regular updates to leadership — all without interrupting the Incident Commander or technical responders. The scribe role (not comms lead) logs actions. Paging additional responders is typically an IC responsibility. Postmortem writing happens after the incident is resolved.
Question 4: A monitoring system sends 500 alerts in a single day during a major outage, most of which are symptoms of one root cause. What monitoring design pattern would BEST reduce this alert storm?
- Alert on symptoms that matter to users (e.g., elevated error rate) rather than on every downstream effect and system metric (Correct answer)
- Implement an alert blackout window during known outage periods
- Increase alert thresholds globally by 50% to reduce sensitivity
- Route all alerts to email instead of paging to reduce interrupt load
Correct answer: Alert on symptoms that matter to users (e.g., elevated error rate) rather than on every downstream effect and system metric
Symptom-based alerting (alert on high user-visible error rates) rather than cause-based alerting (alert on every internal metric deviation) dramatically reduces alert storms by targeting the root signal rather than all downstream effects.
When a root cause affects many subsystems, cause-based monitoring (alerting on CPU, memory, queue depth, connection pools, etc.) generates hundreds of alerts — all symptoms of the same underlying problem. Symptom-based monitoring focuses on user-visible impact: 'error rate above 1%' or 'p99 latency above 2 seconds.' One root cause produces one or two user-visible symptoms, which generate one or two alerts, not hundreds. This is the key insight from Google's SRE book: alert on what breaks users, use dashboards and metrics for diagnosing causes. Blackout windows (option B) suppress useful signals during incidents. Raising all thresholds (option C) reduces sensitivity for future incidents. Routing to email (option D) doesn't reduce the noise, just changes the channel.
Question 5: Which of the following best describes the 'five whys' technique in a postmortem?
- Iteratively asking 'why' about each cause until a systemic root cause is identified, typically requiring around five iterations (Correct answer)
- Identifying five separate contributing factors to an incident
- A formal review panel of five senior engineers who judge the quality of incident response
- Five standard questions asked at the start of every incident to assess severity
Correct answer: Iteratively asking 'why' about each cause until a systemic root cause is identified, typically requiring around five iterations
The five whys is a root cause analysis technique: start with the failure, ask why it happened, then ask why that cause happened, repeating until a fundamental systemic cause is uncovered — typically taking about five iterations.
The five whys, originally developed by Toyota for manufacturing quality control, is widely used in SRE postmortems to avoid stopping analysis at surface-level causes. Example: a service went down (why?) → the server ran out of memory (why?) → a memory leak in the new deployment (why?) → the leak was not caught in testing (why?) → no memory profiling in the CI pipeline (why?) → no policy requiring memory profiling for services with known allocation-heavy code paths. The systemic fix is not just patching the leak, but adding memory profiling to the CI pipeline. The technique's value is identifying organizational and process failures, not just technical ones, leading to actionable, preventive improvements.
Question 6: What is the 'mean time to detect' (MTTD) metric, and why is it important in incident response?
- MTTD is the average time from when an incident starts to when the team becomes aware of it; reducing it limits the window of user impact (Correct answer)
- MTTD is the average time to deploy a permanent fix after an incident is resolved
- MTTD measures how quickly the monitoring system detects configuration drift
- MTTD is the time between two consecutive incidents on the same service
Correct answer: MTTD is the average time from when an incident starts to when the team becomes aware of it; reducing it limits the window of user impact
MTTD measures the detection lag — the time between when a failure occurs and when the on-call team is notified. Lower MTTD means faster response and less user impact during that undetected window.
MTTD (Mean Time to Detect) is one of several key incident metrics alongside MTTR (Mean Time to Repair/Recover) and MTBF (Mean Time Between Failures). MTTD specifically measures the detection gap — if a database started failing at 2:00 AM but the first alert fired at 2:15 AM, MTTD for that incident is 15 minutes. During those 15 minutes, users were experiencing failures without any engineering response. Reducing MTTD requires better monitoring coverage, lower alert thresholds, faster anomaly detection, and synthetic monitoring (probes that actively test user journeys). High MTTD is often caused by alerting only on system metrics (which lag behind user impact) rather than on user-visible SLIs.
SRE Monitoring & Incident Response 2
During a major incident, team members are proposing fixes simultaneously in a group chat, leading to conflicting changes in production.
Which incident management structure BEST prevents this?