SRE Disaster Recovery and Business Continuity 2 — Questions and Answers
Question 1: What is a 'GameDay' exercise in SRE, and what are its primary objectives?
- A GameDay is a planned disaster simulation where the team intentionally induces failure scenarios in a production or production-like environment to test incident response, runbooks, and system resilience (Correct answer)
- A GameDay is a quarterly team-building event where SREs compete to resolve mock incidents the fastest
- A GameDay is an unannounced audit where management tests whether the on-call team can respond to an incident correctly
- A GameDay is a post-mortem review session held after a major incident to replay events and identify improvements
Correct answer: A GameDay is a planned disaster simulation where the team intentionally induces failure scenarios in a production or production-like environment to test incident response, runbooks, and system resilience
GameDays deliberately introduce real failures to validate that the systems, runbooks, monitoring, and people all work together correctly under realistic conditions — finding gaps before real disasters do.
GameDays (pioneered by Amazon, described by Jesse Robbins) are structured exercises where a team deliberately introduces failures — power outages to a rack, terminating a cloud region, deleting a DNS record, filling disk on critical servers — and then responds as if it were a real incident, following runbooks and using real tools. Primary objectives: (1) Validate runbooks against current system state. (2) Identify monitoring gaps (did we detect the failure?). (3) Measure actual RTO vs. assumed RTO. (4) Train team members, especially new ones, in incident response. (5) Build confidence that the system can recover. (6) Find dependencies that aren't documented. (7) Test communication and escalation procedures. Best practices: define a clear blast radius before starting (which services/users can be affected?), have a kill switch to stop the exercise, run in stages (start small, escalate), document everything for a post-exercise review, and track action items from discoveries.
Question 2: In a multi-region cloud deployment, what is 'DNS failover,' and what are its limitations as a DR mechanism?
- DNS failover routes traffic to a secondary region by changing DNS records when the primary region's health check fails; limitations include DNS TTL propagation delay (minutes to hours) and client-side DNS caching that may prevent immediate failover (Correct answer)
- DNS failover is instantaneous because DNS record changes propagate in under one second to all resolvers globally
- DNS failover only works for HTTP-based services; TCP and UDP services require a different failover mechanism
- DNS failover automatically replicates all service state from the primary region to the secondary region upon activation
Correct answer: DNS failover routes traffic to a secondary region by changing DNS records when the primary region's health check fails; limitations include DNS TTL propagation delay (minutes to hours) and client-side DNS caching that may prevent immediate failover
DNS failover is widely used but has inherent delays — TTL values mean resolvers continue sending traffic to the failed primary for up to the TTL duration after the DNS record change, which can range from seconds (low TTL) to hours (high TTL) depending on the configuration.
DNS-based failover works by: (1) Health checks continuously probe the primary endpoint. (2) When the health check fails a threshold, the DNS record is updated to point to the secondary region. (3) Resolvers worldwide cache the old DNS answer until its TTL expires, then fetch the new record. TTL limitations: if TTL is 300 seconds (5 minutes), some clients will continue sending traffic to the failed primary for up to 5 minutes after failover is initiated. If TTL is 3600 seconds (1 hour), clients may be stuck for an hour. Pre-DR preparation: set TTL very low (30-60 seconds) for critical DNS records 24-48 hours before a planned failover, so caches expire quickly. Additional limitations: DNS failover does not handle session state — active user sessions in the primary region are lost. Stateful services need session replication or sticky session fallback. DNS failover is also not suitable for sub-minute RTO requirements. Cloud-managed DNS failover (Route 53, Google Cloud DNS) can achieve reliable failover in 30-60 seconds with low TTL settings.
Question 3: What is the 'shared fate' problem in DR architecture, and how do you design to avoid it?
- Shared fate occurs when the primary and DR system share a common failure domain (same data center, availability zone, cloud account, or dependency) that can take down both simultaneously; avoid by truly isolating DR infrastructure (Correct answer)
- Shared fate is when multiple microservices share the same database, creating a single point of failure for all services
- Shared fate refers to the risk that all team members are traveling together and could be unavailable during a disaster
- Shared fate is a compliance concern about multiple clients sharing the same DR infrastructure
Correct answer: Shared fate occurs when the primary and DR system share a common failure domain (same data center, availability zone, cloud account, or dependency) that can take down both simultaneously; avoid by truly isolating DR infrastructure
If the primary and DR systems share any critical component — the same power feed, same cloud region, same VPC, same DNS provider, or same network path — a failure of that shared component takes down both simultaneously, defeating the DR purpose.
Shared fate (also called shared blast radius) is one of the most common DR architecture failures. Examples: (1) Backup stored in the same AWS region as production — a regional outage takes out both. (2) DR database in a different availability zone but same VPC — a misconfigured security group change blocks both. (3) Primary and DR DNS hosted with the same provider — a DNS provider outage takes down both DNS failover and primary access. (4) DR runbooks stored in a Confluence instance that runs in the same production cluster being recovered. (5) Primary and DR connected through the same BGP peer — a routing issue takes down both. DR isolation requirements: (1) Physically separate power and network. (2) Separate cloud accounts or at minimum separate regions. (3) Separate DNS providers (primary on Route 53, DR on Cloudflare). (4) DR runbooks accessible offline or from a completely separate system. (5) Regular testing that specifically validates the DR path works independently of the primary. The fundamental question: what single failure could take down both primary and DR simultaneously?
Question 4: What is 'chaos engineering for DR validation,' and how does it differ from traditional DR testing?
- Chaos engineering for DR introduces failures in production continuously and unpredictably (like Netflix Chaos Monkey) rather than in scheduled exercises, building continuous confidence that DR mechanisms work under realistic conditions (Correct answer)
- Chaos engineering for DR is the same as a GameDay — both involve intentional failure injection during a scheduled window
- Chaos engineering for DR only tests hardware failure scenarios; traditional DR testing focuses on software and configuration failures
- Chaos engineering for DR is only suitable for stateless microservices; traditional DR testing is required for stateful services
Correct answer: Chaos engineering for DR introduces failures in production continuously and unpredictably (like Netflix Chaos Monkey) rather than in scheduled exercises, building continuous confidence that DR mechanisms work under realistic conditions
Traditional DR testing is periodic and scheduled — teams know it's coming and prepare. Chaos engineering runs continuously and without advance notice (at least to the systems), exposing real-world resilience gaps that scheduled tests often miss because teams anticipate and pre-fix them.
Traditional DR testing (annual failover exercise, quarterly GameDay) has a known limitation: teams often 'cheat' by checking systems, pre-testing scripts, and ensuring everything is in order before the scheduled exercise. This reveals gross failures but misses subtle ones. Chaos engineering for DR (continuous, unannounced) works differently: (1) Automated chaos agents randomly terminate instances, inject latency, or fill disks during business hours without warning. (2) The system must detect, alert, and either self-heal or trigger incident response as if it were a real failure. (3) This surfaces real resilience gaps: monitoring that doesn't catch the failure, runbooks that assume tools are available, failover logic that has a race condition. Netflix's Chaos Kong (regularly fails entire AWS regions) and Chaos Gorilla (simulates availability zone failures) are examples. The maturity progression: start with basic chaos (random instance termination) → advance to controlled failure injection → eventually to continuous automated resilience validation. Both scheduled DR testing and continuous chaos engineering have a place in a mature DR program.
Question 5: A company's e-commerce platform has an RPO of 1 hour and RTO of 30 minutes for the product catalog service. Which backup and recovery architecture BEST meets these requirements?
- Synchronous replication to a warm standby in a second region with automated health-check-triggered failover, plus hourly snapshots to cross-region storage for point-in-time recovery (Correct answer)
- Daily backups to a local NAS with a documented manual restoration procedure taking up to 2 hours
- Continuous replication to a hot standby in the same availability zone with automated failover
- Weekly backups to a tape archive with an estimated restoration time of 4 hours
Correct answer: Synchronous replication to a warm standby in a second region with automated health-check-triggered failover, plus hourly snapshots to cross-region storage for point-in-time recovery
A warm standby with continuous/near-continuous replication meets the 30-minute RTO (standby is pre-warmed, failover is automated). Hourly snapshots meet the 1-hour RPO requirement and provide point-in-time recovery for logical failures.
Matching the architecture to requirements: RTO=30 minutes: the recovery process from decision to serving traffic must complete in 30 minutes. This rules out cold standby (provisioning from scratch takes >30 minutes) and manual procedures (too slow and error-prone). Warm standby with automated health-check-triggered failover meets this: the secondary infrastructure is already running at reduced capacity, failover is triggered automatically when health checks fail, and traffic is redirected within minutes. RPO=1 hour: no more than 1 hour of data can be lost. Synchronous replication (zero data loss) or asynchronous replication (minimal data loss) satisfies this, plus hourly snapshots provide recovery from logical corruption. Same availability zone warm standby (option C) doesn't protect against zone-level failures. Daily backups (option B) violate the 1-hour RPO (24 hours of potential loss) and the 2-hour restoration violates RTO. Weekly tape (option D) fails both requirements dramatically.
Question 6: What is 'data sovereignty' in the context of disaster recovery, and how does it constrain DR architecture for regulated industries?
- Data sovereignty laws require that certain data remain within specific geographic jurisdictions; DR backups and replicas must be stored in compliant locations, potentially preventing the use of the geographically distant DR sites that would otherwise provide the best resilience (Correct answer)
- Data sovereignty refers to the requirement that data owners have the ability to delete their data from all backups upon request
- Data sovereignty is the principle that each microservice owns its own data and other services cannot directly access it
- Data sovereignty is a cloud provider certification that confirms data is stored in geographically isolated fault domains
Correct answer: Data sovereignty laws require that certain data remain within specific geographic jurisdictions; DR backups and replicas must be stored in compliant locations, potentially preventing the use of the geographically distant DR sites that would otherwise provide the best resilience
Laws like GDPR (EU), data localization laws (Russia, China, India), and industry regulations (healthcare, finance) restrict where certain data can be stored — this can force DR to use in-country or in-region backup sites rather than the most geographically distant, resilient locations.
Data sovereignty creates real tensions in DR architecture: Ideal DR: replicate data to a geographically distant region (e.g., US-East + US-West, or EU + APAC) to maximize protection against regional disasters (earthquakes, hurricanes, power grid failures). Data sovereignty constraint: EU personal data cannot leave EU borders without adequate protections. Russian personal data of Russian citizens must stay in Russia. Chinese user data must stay in China. Healthcare data may have HIPAA requirements about where it can be stored. Financial data may have country-specific regulations. Compliance-constrained DR: must use in-country or in-jurisdiction DR sites. For the EU, this means EU-East + EU-West rather than EU + US. This reduces geographic separation (EU availability zones may share some regional risks) and may increase cost (fewer options). DR architects in regulated industries must consult legal and compliance teams before selecting DR regions, and must document the data flows for each DR scenario to ensure compliance is maintained during failover.
SRE Disaster Recovery and Business Continuity 2
What is a 'GameDay' exercise in SRE, and what are its primary objectives?