SRE Alerting Strategy and Noise Reduction 3 — Questions and Answers
Question 1: According to the Google SRE book, which question should every alert be able to answer 'yes' to in order to justify its existence?
- Does this alert correlate with at least one other metric to confirm its accuracy?
- Is this alert actionable—does a human need to do something in response right now? (Correct answer)
- Has this alert fired at least once in production in the past 90 days?
- Does this alert have a corresponding automated remediation script attached?
Correct answer: Is this alert actionable—does a human need to do something in response right now?
The Google SRE book's core alerting principle states that every page-worthy alert must require immediate human action; alerts that don't need a human response right now should be tickets or removed entirely.
Question 2: What is the 'Four Golden Signals' framework for monitoring and alerting in distributed systems?
- Availability, Durability, Performance, and Security metrics for each service
- Latency, Traffic, Errors, and Saturation—the key indicators of service health (Correct answer)
- CPU, Memory, Disk I/O, and Network throughput for infrastructure monitoring
- P50, P95, P99, and P999 latency percentiles for user-facing request performance
Correct answer: Latency, Traffic, Errors, and Saturation—the key indicators of service health
The Four Golden Signals (Latency, Traffic, Errors, Saturation) defined in the Google SRE book provide a minimal but comprehensive framework for alerting on the health of any user-facing service.
Question 3: How does error budget burn rate directly influence alerting strategy?
- Higher burn rates require more granular alert thresholds to capture individual error spikes
- Burn rate determines the appropriate alert window: fast burn triggers short-window alerts; slow burn triggers long-window alerts (Correct answer)
- Error budget consumption automatically adjusts alert severity from warning to critical at 50% depletion
- Burn rate is used to calculate the alert noise ratio and suppress non-critical notifications
Correct answer: Burn rate determines the appropriate alert window: fast burn triggers short-window alerts; slow burn triggers long-window alerts
Fast error budget burn (e.g., 14x over 1 hour) warrants short-window urgent alerts, while slow burn (e.g., 1x over days) warrants longer-window lower-priority alerts—matching alert urgency to actual impact speed.
Question 4: What is a 'phantom alert' in SRE monitoring practice?
- An alert that fires in staging but not in production due to environment configuration differences
- An alert that fires based on stale or delayed metrics and resolves itself before an engineer can respond (Correct answer)
- An alert generated by a monitoring agent that has lost connectivity to its data source
- An alert rule that exists in configuration but is never triggered because its threshold is unreachable
Correct answer: An alert that fires based on stale or delayed metrics and resolves itself before an engineer can respond
Phantom alerts fire due to data pipeline delays or metric staleness, creating false urgency that resolves on its own—they erode trust in the alerting system and contribute to engineers ignoring alerts.
Question 5: What is the recommended strategy for handling alert noise during a known, expected maintenance window?
- Increase alert thresholds globally by 50% to account for expected degradation during maintenance
- Silence or suppress alerts for the affected services for the duration of the maintenance window (Correct answer)
- Route all alerts to a dedicated maintenance queue that engineers review post-maintenance
- Disable monitoring entirely during maintenance to prevent false positive data from skewing SLO calculations
Correct answer: Silence or suppress alerts for the affected services for the duration of the maintenance window
Silencing alerts for known maintenance windows prevents noise that would otherwise desensitize engineers, while ensuring the silence is time-bounded and explicitly tied to the maintenance event for accountability.
Question 6: In a microservices architecture, what alerting anti-pattern does 'alert on every dependency failure' represent?
- Redundant alerting, where multiple teams receive the same notification for a single incident
- Cascading alerts, where a single upstream failure generates dozens of downstream service alerts (Correct answer)
- Threshold drift, where dependency failure rates become normalized and thresholds stop catching real issues
- Alert shadowing, where critical service alerts are hidden behind dependency noise in the notification stream
Correct answer: Cascading alerts, where a single upstream failure generates dozens of downstream service alerts
Alerting on every dependency failure causes cascading alerts where one upstream outage spawns dozens of downstream alerts, overwhelming on-call engineers with noise instead of pointing to the single root cause.
Question 7: What is the benefit of using 'alert ownership' metadata (such as team labels) in an alerting system?
- It enables automatic SLO calculation by attributing alert frequency to specific engineering teams
- It ensures alerts are routed to the correct on-call team with the knowledge to resolve the issue (Correct answer)
- It satisfies regulatory requirements for incident attribution in compliance-heavy industries
- It allows alert thresholds to be dynamically adjusted based on each team's error budget balance
Correct answer: It ensures alerts are routed to the correct on-call team with the knowledge to resolve the issue
Alert ownership metadata ensures pages are routed to the team responsible for that service, reducing response time and preventing confusion about who should be investigating a given alert.
SRE Alerting Strategy and Noise Reduction 3
According to the Google SRE book, which question should every alert be able to answer 'yes' to in order to justify its existence?