SRE Chaos Engineering & Resilience 4 — Questions and Answers
Question 1: What is 'latency injection' used to test in a microservices architecture?
- Whether services can process requests faster than the SLO requires
- How upstream callers handle slow dependencies, including timeout and retry behavior (Correct answer)
- That load balancers distribute traffic evenly across healthy replicas
- Whether caches are warmed before services receive production traffic
Correct answer: How upstream callers handle slow dependencies, including timeout and retry behavior
Latency injection slows downstream calls artificially to verify that clients implement correct timeouts, retries, and fallbacks rather than hanging indefinitely.
Question 2: What is the purpose of an 'abort condition' in a chaos experiment?
- To define the completion criteria that mark an experiment as successful
- To automatically halt the experiment if system health degrades beyond a safe threshold (Correct answer)
- To schedule the experiment to stop at a predetermined time regardless of outcome
- To notify stakeholders when the experiment produces unexpected results
Correct answer: To automatically halt the experiment if system health degrades beyond a safe threshold
Abort conditions (sometimes called 'circuit breakers' for experiments) stop chaos injection automatically when key health metrics exceed acceptable bounds.
Question 3: In the context of SRE, what does 'resilience' fundamentally mean?
- A system that never experiences failures or downtime
- A system's ability to withstand and recover from failures while continuing to serve users acceptably (Correct answer)
- The absence of any single point of failure in the architecture
- A system that can scale horizontally to meet any traffic demand
Correct answer: A system's ability to withstand and recover from failures while continuing to serve users acceptably
Resilience is not about preventing all failures but about ensuring the system degrades gracefully and recovers quickly so user impact is minimized.
Question 4: Which practice involves deliberately taking down entire availability zones in production to test failover?
- Canary releases
- Region evacuation drills (Correct answer)
- Load shedding tests
- Synthetic monitoring
Correct answer: Region evacuation drills
Region or AZ evacuation drills redirect all traffic away from one zone to verify that the remaining zones can absorb the load and failover works as designed.
Question 5: What is 'chaos as code' and what benefit does it provide?
- Writing chaos scenarios in YAML/JSON so experiments are version-controlled, repeatable, and reviewable (Correct answer)
- Using LLMs to generate random failure scenarios without human input
- Deploying chaos agents via serverless functions to minimize infrastructure cost
- Encoding SLO thresholds directly into application source code
Correct answer: Writing chaos scenarios in YAML/JSON so experiments are version-controlled, repeatable, and reviewable
Defining chaos experiments as code (e.g., in Git) enables peer review, change tracking, and consistent reproduction of experiments across environments.
Question 6: Which concept describes an architecture where each microservice has its own database to prevent shared data-layer failures?
- Event sourcing
- Database-per-service pattern (Correct answer)
- CQRS (Command Query Responsibility Segregation)
- Shared schema anti-pattern
Correct answer: Database-per-service pattern
The database-per-service pattern prevents one service's database issues from cascading into failures for other services that would share the same data store.
Question 7: During a chaos experiment, an engineer notices the system's error rate has spiked to 40%. What should they do first?
- Continue the experiment to collect more data points before drawing conclusions
- Trigger the abort condition and restore the system to steady state immediately (Correct answer)
- Increase the blast radius to understand the full failure domain
- File an incident ticket and wait for the on-call engineer to respond
Correct answer: Trigger the abort condition and restore the system to steady state immediately
A 40% error rate far exceeds safe thresholds; the correct action is to halt the experiment and recover the system before further damage occurs.
SRE Chaos Engineering & Resilience 4
What is 'latency injection' used to test in a microservices architecture?