SRE Alerting Strategy and Noise Reduction 3

Question 1

According to the Google SRE book, which question should every alert be able to answer 'yes' to in order to justify its existence?

Accepted Answer

Is this alert actionable—does a human need to do something in response right now?

Answer

Does this alert correlate with at least one other metric to confirm its accuracy?

Answer

Has this alert fired at least once in production in the past 90 days?

Answer

Does this alert have a corresponding automated remediation script attached?

Question 2

What is the 'Four Golden Signals' framework for monitoring and alerting in distributed systems?

Accepted Answer

Latency, Traffic, Errors, and Saturation—the key indicators of service health

Answer

Availability, Durability, Performance, and Security metrics for each service

Answer

CPU, Memory, Disk I/O, and Network throughput for infrastructure monitoring

Answer

P50, P95, P99, and P999 latency percentiles for user-facing request performance

Question 3

How does error budget burn rate directly influence alerting strategy?

Accepted Answer

Burn rate determines the appropriate alert window: fast burn triggers short-window alerts; slow burn triggers long-window alerts

Answer

Higher burn rates require more granular alert thresholds to capture individual error spikes

Answer

Error budget consumption automatically adjusts alert severity from warning to critical at 50% depletion

Answer

Burn rate is used to calculate the alert noise ratio and suppress non-critical notifications

Question 4

What is a 'phantom alert' in SRE monitoring practice?

Accepted Answer

An alert that fires based on stale or delayed metrics and resolves itself before an engineer can respond

Answer

An alert that fires in staging but not in production due to environment configuration differences

Answer

An alert generated by a monitoring agent that has lost connectivity to its data source

Answer

An alert rule that exists in configuration but is never triggered because its threshold is unreachable

Question 5

What is the recommended strategy for handling alert noise during a known, expected maintenance window?

Accepted Answer

Silence or suppress alerts for the affected services for the duration of the maintenance window

Answer

Increase alert thresholds globally by 50% to account for expected degradation during maintenance

Answer

Route all alerts to a dedicated maintenance queue that engineers review post-maintenance

Answer

Disable monitoring entirely during maintenance to prevent false positive data from skewing SLO calculations

Question 6

In a microservices architecture, what alerting anti-pattern does 'alert on every dependency failure' represent?

Accepted Answer

Cascading alerts, where a single upstream failure generates dozens of downstream service alerts

Answer

Redundant alerting, where multiple teams receive the same notification for a single incident

Answer

Threshold drift, where dependency failure rates become normalized and thresholds stop catching real issues

Answer

Alert shadowing, where critical service alerts are hidden behind dependency noise in the notification stream

Question 7

What is the benefit of using 'alert ownership' metadata (such as team labels) in an alerting system?

Accepted Answer

It ensures alerts are routed to the correct on-call team with the knowledge to resolve the issue

Answer

It enables automatic SLO calculation by attributing alert frequency to specific engineering teams

Answer

It satisfies regulatory requirements for incident attribution in compliance-heavy industries

Answer

It allows alert thresholds to be dynamically adjusted based on each team's error budget balance