SRE Distributed Systems Design 2 — Questions and Answers
Question 1: In the CAP theorem, what must a distributed system sacrifice during a network partition?
- Either consistency or availability, depending on the design choice (Correct answer)
- Either partition tolerance or availability, but consistency is always preserved
- Either partition tolerance or consistency, but availability is always preserved
- All three properties simultaneously, making the system temporarily unavailable
Correct answer: Either consistency or availability, depending on the design choice
CAP theorem states that during a network partition (P), a system must choose between consistency (C) — all nodes return the same data — and availability (A) — every request receives a response. P is not optional in real distributed systems.
The CAP theorem, proved by Eric Brewer, states that a distributed system cannot simultaneously guarantee Consistency (every read returns the most recent write), Availability (every request receives a non-error response), and Partition Tolerance (the system continues operating despite message loss or delays between nodes). Since network partitions are a reality in any distributed system, partition tolerance is not truly optional — the practical choice is between CP (prioritize consistency, reject requests when uncertain about state) and AP (prioritize availability, serve potentially stale data during a partition). Examples: HBase and Zookeeper are CP systems; Cassandra and DynamoDB are AP systems by default. SREs must understand this trade-off when designing and operating distributed databases.
Question 2: What is a 'thundering herd' problem in distributed systems, and how is it typically mitigated?
- A large number of clients simultaneously send requests when a cache expires or a service recovers, overloading the backend; mitigated by jitter and staggered retries (Correct answer)
- A cascading failure where one service overloads its upstream dependency; mitigated by circuit breakers
- A situation where too many instances of the same service run simultaneously; mitigated by autoscaling policies
- A DDoS attack pattern targeting the backend directly; mitigated by rate limiting at the edge
Correct answer: A large number of clients simultaneously send requests when a cache expires or a service recovers, overloading the backend; mitigated by jitter and staggered retries
Thundering herd occurs when many clients simultaneously stampede a backend — e.g., after a cache miss or service restart — overwhelming it. Jitter (random delays), exponential backoff, and probabilistic cache refresh mitigate it.
Classic thundering herd scenarios: (1) A heavily-shared cache key expires — all clients simultaneously miss the cache and hit the database. (2) A service restarts and all connection pool clients reconnect at the same instant. (3) All clients have the same retry timeout and fire simultaneously after a brief outage. Mitigations: adding random jitter to cache TTLs so keys expire at different times, using probabilistic early expiration (refresh cache slightly before it expires), implementing exponential backoff with jitter for retries, using a cache stampede lock (only one client refills the cache while others wait), and connection pool warmup during restarts. The jitter pattern is broadly applicable — anywhere clients synchronize on a timer, introducing randomness prevents the spike.
Question 3: A microservice calls a downstream service that has become slow, causing the caller's connection pool to fill up. This eventually cascades to the caller's callers. What design pattern would have PREVENTED this cascade?
- Circuit breaker pattern — automatically stop sending requests to the slow downstream before resource exhaustion occurs (Correct answer)
- Retry with exponential backoff — keep retrying the slow downstream until it recovers
- Load balancer health checks — remove the slow instance from the pool
- Service mesh mTLS — encrypt traffic to prevent slowdowns caused by security overhead
Correct answer: Circuit breaker pattern — automatically stop sending requests to the slow downstream before resource exhaustion occurs
The circuit breaker pattern detects that a downstream is slow or failing and temporarily stops sending requests to it, preventing the caller's thread pool and connection pool from exhausting and cascading the failure upward.
The circuit breaker pattern, popularized by Michael Nygard's 'Release It!' and used in libraries like Hystrix and Resilience4j, works in three states: Closed (normal operation, requests flow), Open (downstream is failing, requests fail immediately without attempting the call), and Half-Open (periodic probe requests to check recovery). When a slow downstream holds connections, a circuit breaker opens and causes the caller to fail fast with a fallback instead of waiting for timeouts. This prevents thread/connection pool exhaustion. Retries with backoff (option B) would make the problem worse — more retries into a slow service. Health checks (option C) help but only remove completely failed instances, not slow ones. mTLS (option D) is irrelevant to this failure mode.
Question 4: In a distributed system using eventual consistency, a user updates their profile picture and immediately refreshes the page, but sees their old picture. What is the MOST accurate explanation for this behavior?
- The read was served from a replica that had not yet received the write replication, which is expected behavior in an eventually consistent system (Correct answer)
- The user's write failed silently and the profile picture was never updated
- The CDN cached the old image and is serving it despite the database being updated
- The database had a network partition and both the read and write were served from different partitioned nodes
Correct answer: The read was served from a replica that had not yet received the write replication, which is expected behavior in an eventually consistent system
In eventually consistent systems, reads from replicas may temporarily return stale data because replication is asynchronous and the replica may not have received the write yet. This is expected, not an error.
Eventual consistency is a consistency model used by many distributed databases (DynamoDB, Cassandra, Riak) that guarantees all replicas will eventually converge to the same value, but reads may temporarily return stale data immediately after a write. This is not a bug — it is the defined behavior of the system. The trade-off is higher availability and lower write latency (writes do not need to wait for all replicas to acknowledge). For user-facing features where users expect to immediately see their own changes, systems often implement 'read your own writes' consistency (route the user's reads to the same replica they wrote to, or use a more consistent read mode for their session). CDN caching (option C) is a separate issue, and a network partition (option D) would likely produce errors, not stale reads.
Question 5: What is the purpose of a 'sidecar proxy' in a service mesh architecture?
- To handle cross-cutting concerns like load balancing, circuit breaking, observability, and mTLS on behalf of the service without modifying the service code (Correct answer)
- To provide a redundant backup instance that takes over if the main service container crashes
- To act as an API gateway that routes external traffic to the correct microservices
- To cache database query results locally to reduce backend load
Correct answer: To handle cross-cutting concerns like load balancing, circuit breaking, observability, and mTLS on behalf of the service without modifying the service code
A sidecar proxy (e.g., Envoy in Istio) runs alongside each service instance and intercepts all network traffic, transparently applying retries, circuit breaking, mutual TLS, distributed tracing, and load balancing without any changes to the service code.
The sidecar pattern in service meshes (Istio/Envoy, Linkerd, Consul Connect) deploys a proxy container alongside every service instance in the same pod or VM. All inbound and outbound traffic flows through the sidecar, which can enforce: mTLS for service-to-service authentication, automatic retries and circuit breaking, load balancing across upstream instances, distributed tracing by injecting trace headers, and traffic management policies. The crucial advantage is that these behaviors are consistent across all services regardless of programming language, and service teams do not need to implement networking logic in their application code. An API gateway (option C) handles north-south (external-to-cluster) traffic, not east-west (service-to-service) traffic.
Question 6: Which consistency model is required for a distributed counter that tracks the total number of page views across multiple servers, where slight temporary inaccuracies are acceptable but the count must eventually be correct?
- Eventual consistency with a CRDT (Conflict-free Replicated Data Type) counter (Correct answer)
- Strong consistency requiring a distributed lock on every increment
- Read-your-writes consistency so each server sees its own increments immediately
- Linearizability so all servers agree on the counter value after every operation
Correct answer: Eventual consistency with a CRDT (Conflict-free Replicated Data Type) counter
A CRDT increment counter (G-Counter) supports commutative, associative merges — each server tracks its own count and values are merged to get the global total. This provides eventual consistency without coordination on each write.
CRDTs (Conflict-free Replicated Data Types) are data structures mathematically designed for eventual consistency. A G-Counter (Grow-only Counter) assigns each server a slot in a vector: each server increments only its own slot, and the global count is the sum of all slots. Servers can merge vectors by taking the maximum of each slot. This operation is commutative (order doesn't matter), associative (grouping doesn't matter), and idempotent (applying the same update twice gives the same result) — so replicas can merge without coordination or locks, and eventually converge to the same total. Strong consistency (option B) requires coordination on every write, making it expensive and availability-reducing at scale. Linearizability (option D) is even stricter and impractical for a high-throughput page view counter.
SRE Distributed Systems Design 2
In the CAP theorem, what must a distributed system sacrifice during a network partition?