SRE Change Management & Postmortem Practices 2 — Questions and Answers
Question 1: A change management policy requires a 48-hour review window for all production changes. An SRE argues this policy creates more risk than it reduces. What is the BEST argument supporting the SRE's position?
- Long review windows delay security patches and bug fixes, creating a larger window of vulnerability, and may incentivize engineers to batch changes in ways that increase blast radius (Correct answer)
- Engineers will simply bypass the review process if it is too slow, making it ineffective
- 48-hour reviews are only necessary for changes to customer-facing services, not internal infrastructure
- Review processes should be automated and do not need human review windows
Correct answer: Long review windows delay security patches and bug fixes, creating a larger window of vulnerability, and may incentivize engineers to batch changes in ways that increase blast radius
Fixed delay review windows create perverse incentives: engineers batch many changes together to minimize review overhead, creating larger, riskier deployments. They also delay critical security and stability fixes that need to ship immediately.
Research by the DORA (DevOps Research and Assessment) team shows that high change frequency with small batch sizes reduces deployment risk more than large, infrequent deployments with heavy review gates. When a 48-hour review is required for every change, engineers rationally batch multiple changes into single deployments to reduce review overhead. This creates larger, harder-to-diagnose deployment risks. Emergency security patches may be delayed by 48 hours, leaving systems vulnerable. A better approach: automated testing and validation gates (immediate, no delay), risk-tiered human review (high-risk changes get more scrutiny, low-risk changes get automated approval), and emergency procedures that allow critical fixes to bypass normal windows with post-hoc review.
Question 2: What is 'change freeze' and when is it MOST appropriately used?
- A temporary moratorium on production changes during high-traffic periods (e.g., holiday season) or when the error budget is exhausted, to minimize deployment risk (Correct answer)
- A permanent policy that prevents any changes to stable production services regardless of business need
- A tool that automatically reverts all changes made in the last 24 hours when an incident occurs
- A code review process that requires two approvals before any change reaches production
Correct answer: A temporary moratorium on production changes during high-traffic periods (e.g., holiday season) or when the error budget is exhausted, to minimize deployment risk
Change freezes are temporary policies applied during periods of elevated risk — high-traffic events, exhausted error budgets, or immediately after a major incident — to prevent deployments from introducing additional failures during a sensitive period.
Change freezes are a risk management tool used when the cost of a bad deployment is highest. Common triggers: (1) Holiday or peak traffic periods when traffic is 10× normal and any degradation has maximum customer impact. (2) Error budget exhaustion — no remaining budget to absorb deployment risk. (3) After a major incident where the system is recovering and stability is uncertain. (4) Before major business events (product launches, earnings calls). Change freezes should be time-bounded, scope-defined (which services are frozen), and have clear exception procedures for emergency security fixes. Permanent freezes (option B) would prevent all improvements and security patches. The automatic revert description (option C) is a rollback mechanism, not a freeze. Option D describes a standard code review process.
Question 3: In a blameless postmortem, a root cause is identified as 'engineer fatigue from excessive on-call load.' What category of action item BEST addresses this root cause?
- Organizational: reduce on-call load through automation, better runbooks, and alert tuning to address the systemic overwork condition (Correct answer)
- Technical: deploy additional monitoring to detect when engineers are fatigued
- Process: require on-call engineers to confirm they are well-rested before each shift
- Individual: counsel the engineer on time management and self-care practices
Correct answer: Organizational: reduce on-call load through automation, better runbooks, and alert tuning to address the systemic overwork condition
Engineer fatigue is an organizational and systemic problem — the on-call load is too high. The fix is organizational: reduce the operational burden through automation, better alerting, and improved runbooks so the load is sustainable.
Blameless postmortem analysis treats human error as a symptom of an underlying systemic condition, not the root cause itself. If an engineer made a mistake because they were fatigued from an unsustainable on-call rotation, the root cause is the unsustainable on-call load, not the individual's behavior. Appropriate systemic fixes include: reducing unnecessary alert noise (the most common contributor to on-call fatigue), creating better runbooks so incidents resolve faster, automating common remediation steps, and improving the on-call rotation to ensure adequate rest time. Monitoring engineer fatigue (option B) detects it but doesn't fix it. Requiring self-declaration (option C) adds process burden to an already overloaded engineer. Individual counseling (option D) attributes a systemic problem to personal inadequacy, the opposite of blameless culture.
Question 4: A postmortem action item reads: 'Engineers should be more careful when modifying firewall rules.' Why is this action item considered POOR quality?
- It is vague, non-measurable, places blame on individuals, and will not prevent recurrence — a good action item specifies a concrete technical or process change (Correct answer)
- Firewall rules should not be in scope for an application-level postmortem
- The action item is too short and needs more technical detail about which specific rules need attention
- Action items should always be assigned to the SRE team, not individual engineers
Correct answer: It is vague, non-measurable, places blame on individuals, and will not prevent recurrence — a good action item specifies a concrete technical or process change
Telling engineers to 'be more careful' is a non-actionable, unmeasurable request that relies on human behavior change rather than systemic improvement. It will not prevent the same mistake in a different context or by a different person.
High-quality postmortem action items are SMART: Specific (what exactly will be done?), Measurable (how will we know it's done?), Assignable (who owns it?), Relevant (does it prevent recurrence?), and Time-bound (when will it be completed?). 'Be more careful' fails on all dimensions. Better action items for a firewall rule mistake might include: 'Implement automated validation that runs lint and connectivity tests on all firewall rule changes before merge (Owner: Platform team, Due: 2026-05-01)' or 'Add firewall changes to the high-risk change category requiring two-person review (Owner: Change Management, Due: 2026-04-15).' These are testable, have clear owners, and will prevent the class of failure regardless of which engineer is involved.
Question 5: What is the recommended time limit for completing a postmortem draft after an incident?
- Within 24–48 hours while details are fresh, with a final review within 5 business days (Correct answer)
- Within one hour of incident resolution to capture real-time observations
- Within 30 days to allow sufficient time for a thorough investigation
- There is no standard — postmortems should be completed whenever the team has capacity
Correct answer: Within 24–48 hours while details are fresh, with a final review within 5 business days
Postmortems should be drafted within 24–48 hours to capture accurate incident details while memory is fresh, then finalized with review and action item assignment within a week.
Google's SRE practice recommends that postmortem drafts be created within 24–72 hours of incident resolution. The reasons: memory of specific actions, decisions, and observations fades quickly; log retention windows may expire; and urgency in identifying and assigning action items declines sharply over time. Within one hour (option B) is too fast for a thorough analysis — the immediate focus should be stabilization and monitoring. 30 days (option C) is too late — details are lost, engineers may have moved to other projects, and the organizational learning opportunity is missed. 'When we have capacity' (option D) typically means postmortems never get completed for lower-severity incidents. Best practice: assign a postmortem owner immediately when the incident is resolved, with a deadline.
Question 6: What is the purpose of a 'production readiness review' (PRR) before a new service launches?
- To ensure the service meets reliability, scalability, and operability standards before it enters production, preventing the SRE team from inheriting an unmaintainable service (Correct answer)
- To assess the financial cost of running the service in production over its expected lifetime
- To verify that the service's features meet the product requirements defined in the original specification
- To conduct a security audit of the service's code for vulnerabilities before launch
Correct answer: To ensure the service meets reliability, scalability, and operability standards before it enters production, preventing the SRE team from inheriting an unmaintainable service
A PRR ensures a new service is production-ready from a reliability and operability perspective — it has monitoring, runbooks, alerting, capacity planning, and meets SRE team standards before the team takes on operational responsibility.
The Production Readiness Review (PRR) is a formal gate in Google's SRE model that determines whether a service is ready to be onboarded by an SRE team. It assesses: Does the service have meaningful monitoring and alerting? Is there an on-call runbook? Has the service been load-tested? Is the capacity plan reviewed? Does the service have a defined SLO? Are there automated rollback procedures? Is the architecture reviewed for known reliability antipatterns? The PRR protects both the service (preventing premature production exposure) and the SRE team (preventing them from inheriting a service that will generate excessive toil). Services that fail a PRR return to the development team with a list of required improvements before SRE support is granted. It is not a feature review (option C) or a security audit (option D), though security may be one dimension.
SRE Change Management & Postmortem Practices 2
A change management policy requires a 48-hour review window for all production changes.
An SRE argues this policy creates more risk than it reduces.
What is the BEST argument supporting the SRE's position?