SRE Chaos Engineering & Resilience 3 — Questions and Answers
Question 1: What is 'network partition testing' designed to validate in a distributed system?
- That the network can handle peak throughput without packet loss
- That services handle split-brain scenarios and communication loss between nodes gracefully (Correct answer)
- That DNS resolution works correctly under high query load
- That TLS certificates renew automatically before expiration
Correct answer: That services handle split-brain scenarios and communication loss between nodes gracefully
Network partition tests verify that distributed services handle scenarios where nodes cannot communicate, preventing data inconsistency or full outages.
Question 2: Which chaos engineering principle states that experiments should be 'scientific' with a defined hypothesis?
- Minimal footprint principle
- Hypothesis-driven experimentation principle (Correct answer)
- Chaos as code principle
- Automate everything principle
Correct answer: Hypothesis-driven experimentation principle
Chaos engineering follows the scientific method: form a hypothesis about system behavior, inject failure, and measure whether the hypothesis holds.
Question 3: What is a 'runbook' and why is it critical to resilience during chaos experiments?
- A monitoring dashboard that tracks experiment metrics in real time
- A documented set of procedures for responding to and recovering from specific failure scenarios (Correct answer)
- A script that automatically terminates misbehaving services
- A log aggregation pipeline that collects chaos experiment output
Correct answer: A documented set of procedures for responding to and recovering from specific failure scenarios
Runbooks provide step-by-step recovery instructions so on-call engineers can respond consistently and quickly when chaos experiments reveal failure modes.
Question 4: In resilience architecture, what does 'bulkhead isolation' prevent?
- Cascading failures from one service component spreading to unrelated components (Correct answer)
- Unauthorized access to sensitive microservices via lateral movement
- Database connection exhaustion caused by slow queries
- CPU contention between co-located containers on the same node
Correct answer: Cascading failures from one service component spreading to unrelated components
Bulkhead isolation (borrowed from ship hull design) partitions resources so that a failure in one pool cannot exhaust resources needed by another.
Question 5: What does 'mean time to recovery' (MTTR) measure in the context of chaos engineering outcomes?
- The average time between successive failures in a system
- The average time taken to restore normal service after a failure is detected (Correct answer)
- The percentage of time a system is available over a rolling 30-day window
- The frequency at which chaos experiments are scheduled per month
Correct answer: The average time taken to restore normal service after a failure is detected
MTTR measures how quickly a team can restore service after an incident, and chaos engineering aims to reduce it by improving detection and recovery procedures.
Question 6: Which tool is commonly used to inject chaos into Kubernetes workloads using CRDs (Custom Resource Definitions)?
- Prometheus
- LitmusChaos (Correct answer)
- Grafana
- Argo CD
Correct answer: LitmusChaos
LitmusChaos uses Kubernetes-native CRDs and operators to define, schedule, and execute chaos experiments on pods, nodes, and network paths.
Question 7: What is the difference between 'proactive' and 'reactive' resilience practices?
- Proactive practices fix production outages; reactive practices prevent them
- Proactive practices identify and fix weaknesses before incidents occur; reactive practices respond after failures happen (Correct answer)
- Proactive practices use manual processes; reactive practices use automated tooling
- Proactive practices apply to stateless services; reactive practices apply to stateful databases
Correct answer: Proactive practices identify and fix weaknesses before incidents occur; reactive practices respond after failures happen
Proactive resilience (chaos experiments, DR drills) finds weaknesses before users are impacted; reactive resilience (incident response, postmortems) handles failures after they occur.
SRE Chaos Engineering & Resilience 3
What is 'network partition testing' designed to validate in a distributed system?