SRE Chaos Engineering & Resilience 2 — Questions and Answers
Question 1: What is the primary purpose of a 'steady state hypothesis' in chaos engineering?
- To document the normal behavior of a system before introducing failures (Correct answer)
- To predict which components will fail during an experiment
- To define the maximum acceptable downtime for a service
- To establish SLO thresholds for production traffic
Correct answer: To document the normal behavior of a system before introducing failures
The steady state hypothesis defines observable, measurable normal behavior so you can verify the system returns to that state after chaos is injected.
Question 2: Which blast radius control technique limits a chaos experiment to only 10% of production pods?
- Canary deployment strategy
- Percentage-based targeting with a selector (Correct answer)
- Blue-green traffic splitting
- Feature flag gating
Correct answer: Percentage-based targeting with a selector
Percentage-based pod selectors (e.g., in LitmusChaos or Chaos Monkey) restrict the experiment scope to a fraction of the fleet.
Question 3: What does 'fault injection' mean in the context of resilience testing?
- Deploying broken code to staging environments
- Deliberately introducing errors, latency, or resource exhaustion into a running system (Correct answer)
- Scanning source code for potential runtime exceptions
- Blocking all external traffic to simulate a DDoS event
Correct answer: Deliberately introducing errors, latency, or resource exhaustion into a running system
Fault injection intentionally introduces controlled failures (network drops, CPU spikes, disk full) to observe how the system responds.
Question 4: In GameDay exercises, what is the role of the 'chaos team' versus the 'ops team'?
- The chaos team deploys new features while ops monitors metrics
- The chaos team injects failures while the ops team detects and responds as they would in a real incident (Correct answer)
- The chaos team reviews postmortems while ops executes runbooks
- The chaos team manages the load balancer while ops handles DNS
Correct answer: The chaos team injects failures while the ops team detects and responds as they would in a real incident
GameDays simulate real incidents: one group injects faults while responders practice detection and recovery under realistic conditions.
Question 5: Which metric best indicates that a system has recovered from a chaos experiment?
- CPU utilization returning to baseline
- Error budget consumption stopping
- All steady state indicators returning to pre-experiment values (Correct answer)
- The number of alerts generated dropping to zero
Correct answer: All steady state indicators returning to pre-experiment values
Recovery is confirmed when all predefined steady state observables (latency, error rate, throughput) return to their normal ranges.
Question 6: What is 'Chaos Monkey' and which company originally developed it?
- A load testing tool developed by Google
- A random instance terminator developed by Netflix as part of the Simian Army (Correct answer)
- A network packet dropper developed by Amazon
- A memory leak injector developed by Microsoft
Correct answer: A random instance terminator developed by Netflix as part of the Simian Army
Netflix created Chaos Monkey to randomly terminate EC2 instances in production, forcing engineers to build resilient auto-recovering services.
Question 7: When should chaos experiments first be run in a CI/CD pipeline?
- Only in production after full deployment
- In staging or pre-production environments before changes reach production (Correct answer)
- Exclusively in development environments on developer laptops
- Only during scheduled maintenance windows in production
Correct answer: In staging or pre-production environments before changes reach production
Running chaos experiments in staging catches resilience regressions before they reach production, shifting reliability testing left in the pipeline.
SRE Chaos Engineering & Resilience 2
What is the primary purpose of a 'steady state hypothesis' in chaos engineering?