SRE Disaster Recovery and Business Continuity 1 — Questions and Answers
Question 1: What is the difference between RTO (Recovery Time Objective) and RPO (Recovery Point Objective)?
- RTO is the maximum acceptable time to restore service after a disaster; RPO is the maximum acceptable amount of data loss measured in time (how old the most recent backup can be) (Correct answer)
- RTO measures how quickly the team is alerted to a disaster; RPO measures how quickly the first response team arrives on-site
- RTO is the time required to restore from backup; RPO is the time required to validate data integrity after restoration
- Both RTO and RPO measure recovery time; RTO applies to primary systems and RPO applies to secondary systems
Correct answer: RTO is the maximum acceptable time to restore service after a disaster; RPO is the maximum acceptable amount of data loss measured in time (how old the most recent backup can be)
RTO (Recovery Time Objective) defines how long the business can tolerate being down. RPO (Recovery Point Objective) defines how much data loss is acceptable — if RPO is 1 hour, backups must occur at least every hour.
RTO and RPO are the two fundamental DR metrics that define the recovery requirements a business can accept: RTO — Maximum acceptable downtime after a disaster declaration. If RTO is 4 hours, the service must be restored and serving users within 4 hours. This drives architectural decisions about standby capacity (warm standby is faster but more expensive; cold standby is cheaper but slower). RPO — Maximum acceptable data loss, expressed as a time window. If RPO is 15 minutes, data older than 15 minutes must never be permanently lost. This drives backup frequency — a 15-minute RPO requires continuous replication or backups every 15 minutes. Key relationship: lower RTO and RPO require more investment (continuous replication, active-active architecture, hot standby). Higher RTO and RPO allow lower-cost strategies (daily backups, cold standby). These must be business-driven, not technically arbitrary.
Question 2: What is an 'active-active' disaster recovery architecture, and how does it differ from 'active-passive'?
- Active-active runs identical service instances in multiple regions simultaneously, all serving traffic; active-passive keeps a standby region that is not serving traffic until failover is triggered (Correct answer)
- Active-active uses two identical servers behind a load balancer in a single data center; active-passive uses geographically distributed servers
- Active-active requires full data synchronization between all regions; active-passive only synchronizes configuration files
- Active-active is only appropriate for stateless services; active-passive is only appropriate for stateful services
Correct answer: Active-active runs identical service instances in multiple regions simultaneously, all serving traffic; active-passive keeps a standby region that is not serving traffic until failover is triggered
Active-active (multi-region live traffic) provides zero-downtime failover and distributes load, but requires conflict resolution for distributed writes. Active-passive (standby warm/cold) is simpler and cheaper but has failover time proportional to how 'warm' the standby is.
Active-Active: All regions (typically 2-3) handle live production traffic simultaneously. Load balancers distribute requests across regions. If one region fails, the other(s) absorb the traffic — typically zero or near-zero RTO. Write conflicts must be handled (last-write-wins, CRDT, or application-level resolution). Higher cost (full capacity in each region). Examples: global CDN-fronted services, multi-region DynamoDB Global Tables. Active-Passive (Warm Standby): Primary region serves all traffic. Secondary region runs at reduced capacity (pre-warmed), ready to scale up. Failover: redirect DNS to secondary (RTO: minutes). Data replication lag = RPO. Active-Passive (Cold Standby): Secondary region has infrastructure provisioned but not running (IaC templates, AMIs). Failover: provision and start all resources (RTO: 15-60 minutes). Lowest cost but highest recovery time. Choice depends on RTO/RPO requirements and cost tolerance.
Question 3: Why is it critical to regularly TEST disaster recovery procedures rather than just documenting them?
- DR procedures drift from reality as systems change — untested plans frequently fail during actual disasters, and regular testing reveals gaps, validates RTOs, and ensures the team has practiced the steps (Correct answer)
- DR testing is primarily a compliance requirement and provides limited practical benefit beyond audit documentation
- DR procedures written by experienced engineers are reliable without testing; testing is only needed for procedures written by junior staff
- Testing is only needed when the DR procedure was most recently updated more than 12 months ago
Correct answer: DR procedures drift from reality as systems change — untested plans frequently fail during actual disasters, and regular testing reveals gaps, validates RTOs, and ensures the team has practiced the steps
Systems change continuously. DR procedures that weren't tested against the current production environment often fail — a backup that wasn't monitored may be corrupt, a failover script may reference decommissioned infrastructure, or the RTO assumption may have been unrealistic.
Untested DR plans have a well-documented track record of failure during actual disasters. Common failures: (1) Backup corruption or incomplete backups that were never validated with a restore test. (2) Runbook commands referencing decommissioned servers, old credential names, or deprecated tools. (3) RTO estimates that were guesses, not measurements — the actual restore takes 4 hours, not the planned 30 minutes. (4) Team members who wrote the runbook have left; new team members have never practiced the steps. (5) Database schema changes since the last backup make the backup incompatible with the current application. DR testing best practices: Game Days (regularly scheduled full DR simulations), chaos engineering DR scenarios (intentionally fail a region), annual or semi-annual full failover tests with real traffic, continuous backup validation (automated restore-to-staging tests), and runbook reviews whenever major system changes are made.
Question 4: A company's DR plan declares a 99.99% availability SLA. Their current backup strategy is daily backups to a single S3 bucket in the same AWS region. What is the MOST critical gap in this plan?
- Daily backups create an RPO of up to 24 hours, and a single-region backup does not protect against a regional AWS outage — both are incompatible with 99.99% availability (Correct answer)
- S3 is not a reliable backup target; the company should use a dedicated backup appliance
- The backup frequency is adequate; the gap is that the company has not documented the restoration procedure
- The SLA only needs backups if the company has signed contracts with financial penalties for downtime
Correct answer: Daily backups create an RPO of up to 24 hours, and a single-region backup does not protect against a regional AWS outage — both are incompatible with 99.99% availability
99.99% availability ≈ 52 minutes of downtime per year. A 24-hour RPO and single-region backup is incompatible — a regional failure would cause data loss of up to 24 hours and recovery time far exceeding 52 minutes.
99.99% availability means the service can be unavailable for at most 0.01% × 365 × 24 × 60 ≈ 52.6 minutes per year. A daily backup strategy means: RPO = up to 24 hours of potential data loss (unacceptable for most services targeting 99.99%). Recovery from a full backup typically takes hours (restore database, validate integrity, update application configuration, redirect traffic) — likely exceeding the 52-minute annual budget in a single incident. Single-region backup: if the AWS region hosting both production and backups experiences an outage (which AWS regions have, historically), the backup is also unavailable, making recovery impossible until the region recovers. Required improvements: continuous or near-continuous replication (RPO of seconds to minutes), cross-region backup replication, automated failover to a standby region, and tested recovery procedures with measured RTO well under 52 minutes.
Question 5: What is a 'runbook' in the context of disaster recovery, and what makes it operationally effective?
- A runbook is a documented, step-by-step procedure for responding to specific failure scenarios; it is effective when it is concrete (exact commands), tested, maintained, and executable by someone unfamiliar with the system (Correct answer)
- A runbook is a high-level architecture document describing the recovery architecture for compliance audits
- A runbook is an automated script that runs without human intervention during disaster recovery
- A runbook is a post-incident summary document that captures what was done during the recovery for future reference
Correct answer: A runbook is a documented, step-by-step procedure for responding to specific failure scenarios; it is effective when it is concrete (exact commands), tested, maintained, and executable by someone unfamiliar with the system
An effective runbook contains exact commands (not just concepts), has been tested in a real or realistic environment, is kept up to date with system changes, and can be followed by an on-call engineer who did not write it — typically at 3 AM under stress.
Runbooks bridge the gap between 'we have a DR plan' and 'we can actually recover under pressure.' Characteristics of effective runbooks: (1) Specific commands: not 'restart the database' but the exact kubectl/service command with expected output and error handling. (2) Prerequisites stated: what access, credentials, and tools are needed before starting. (3) Decision points: if X, do Y; if Z, do W — the runbook handles realistic variations. (4) Verification steps: after each action, how to confirm it worked. (5) Rollback instructions: how to undo the action if it makes things worse. (6) Tested: the runbook has been executed end-to-end in a realistic environment (DR drill, GameDay), not just reviewed on paper. (7) Maintained: updated whenever the system it describes changes. (8) Accessible: stored in a location that is available when production is down (not in a Confluence instance that's hosted on the same infrastructure that's failing).
Question 6: What is the difference between a 'backup' and 'replication' as data protection strategies?
- Backups are point-in-time snapshots that protect against accidental deletion and data corruption; replication copies live data to another location in near real-time and protects against site failures but not data corruption (Correct answer)
- Backups are stored in the same data center as production; replication always occurs to a geographically distant location
- Backups are immutable; replication creates mutable copies that can be overwritten by the primary
- Backups protect databases only; replication is used for file systems only
Correct answer: Backups are point-in-time snapshots that protect against accidental deletion and data corruption; replication copies live data to another location in near real-time and protects against site failures but not data corruption
Replication mirrors live data near-instantly (low RPO) but cannot protect against data corruption or accidental deletion since the corruption is replicated too. Backups capture a known-good state at a point in time, providing recovery from logical failures, but have higher RPO.
The distinction between backup and replication is critical for DR design: Replication (synchronous or asynchronous): continuously mirrors all writes from primary to secondary. RPO: seconds (async) to near-zero (sync). Protects against: hardware failure, data center outage, network failures. Does NOT protect against: accidental data deletion (DELETE propagated to replica), data corruption (corrupted data propagated to replica), ransomware encryption (encryption propagated to replica), application bugs that corrupt data. Backups (point-in-time snapshots): capture the state of data at specific moments (hourly, daily). RPO: time since last backup. Protects against: all replication failure modes above — if you need data from 3 days ago before a corruption event, only a backup can provide it. Does NOT protect against: failures occurring between backup intervals. Complete DR strategy requires BOTH: replication for low-RPO protection against infrastructure failures, AND backups for logical failure protection. Backup retention policy (how many backups to keep) determines how far back recovery is possible.
SRE Disaster Recovery and Business Continuity 1
What is the difference between RTO (Recovery Time Objective) and RPO (Recovery Point Objective)?