SRE Performance Testing and Load Management 1 — Questions and Answers
Question 1: What is the difference between a 'load test' and a 'stress test,' and when should each be used?
- A load test validates performance under expected production traffic levels to verify SLOs are met; a stress test pushes beyond expected levels to find breaking points and understand failure modes (Correct answer)
- A load test runs for a short duration (minutes); a stress test runs for a long duration (hours or days) to detect memory leaks and gradual degradation
- A load test uses synthetic traffic; a stress test uses replayed production traffic to simulate realistic user behavior
- A load test applies constant traffic; a stress test ramps traffic exponentially to simulate viral growth scenarios
Correct answer: A load test validates performance under expected production traffic levels to verify SLOs are met; a stress test pushes beyond expected levels to find breaking points and understand failure modes
Load tests confirm that the system handles expected traffic volumes while meeting SLOs. Stress tests find capacity limits and failure behaviors by pushing beyond expected volumes — revealing what breaks first and how the system fails.
Performance test taxonomy: Load test: applies traffic at or near the expected production level (e.g., average daily peak). Goal: confirm latency SLOs are met, verify no memory leaks under sustained load, validate autoscaling responds correctly. Run before major releases and quarterly. Stress test: applies traffic well above expected production levels (e.g., 2-5× peak) to find the system's breaking point. Goal: identify the first bottleneck to saturate, understand failure behavior (graceful degradation vs. cascading failure), determine safe operating headroom. Run before capacity planning cycles and before high-traffic events. Soak test (endurance test): applies moderate load for an extended period (hours to days). Goal: find slow memory leaks, connection pool exhaustion over time, disk fill from logs. Spike test: applies sudden, sharp traffic increases. Goal: validate autoscaling speed and behavior during viral events or flash sales. Each type answers different reliability questions and should be part of a comprehensive performance testing program.
Question 2: What is 'throughput' vs. 'latency' in performance testing, and what is their typical trade-off relationship?
- Throughput is the number of requests processed per second; latency is the time to process a single request. At low load, both can be optimized simultaneously, but at high load, higher throughput often comes with increased latency as queuing occurs (Correct answer)
- Throughput and latency are inversely proportional at all load levels — any increase in throughput always causes a proportional increase in latency
- Throughput measures front-end performance; latency measures back-end database performance — they are independent metrics
- Throughput and latency are the same metric expressed in different units — higher throughput always means lower latency
Correct answer: Throughput is the number of requests processed per second; latency is the time to process a single request. At low load, both can be optimized simultaneously, but at high load, higher throughput often comes with increased latency as queuing occurs
At low utilization, adding more requests doesn't significantly increase individual request latency. As utilization approaches capacity, queueing theory (Little's Law) predicts latency increases sharply — this is the hockey-stick latency curve observed in most systems.
The throughput/latency relationship is described by queueing theory, particularly Little's Law (L = λW, where L = number in system, λ = arrival rate, W = wait time). At low utilization (e.g., 20-50% of capacity): adding more requests doesn't significantly increase queuing; latency is dominated by service time (processing time), not wait time; both throughput and latency can be good simultaneously. As utilization approaches 80-90%: queues begin to form; latency increases significantly even as throughput plateaus. Near 100% utilization: latency becomes extremely high (unbounded queuing); adding more requests doesn't increase throughput but dramatically increases latency. The hockey-stick curve: latency stays flat until ~70-80% utilization, then rises sharply. This is why performance SLOs should be validated at realistic production utilization levels, not at 10% utilization where everything looks fast. Implications for capacity planning: to maintain SLOs, systems should be sized so that peak production traffic uses no more than 60-70% of capacity, leaving headroom for traffic spikes before latency degrades.
Question 3: What is a 'p99 latency' measurement, and why do SREs prefer it over average (mean) latency for SLO monitoring?
- p99 is the 99th percentile latency — 99% of requests complete faster than this value; it captures the experience of the worst-served 1% of users, which average latency hides by diluting outliers in the mean (Correct answer)
- p99 latency is calculated as 99% of the maximum observed latency, providing a conservative upper bound for capacity planning
- p99 latency measures latency for premium (99th tier) users who receive priority service, reflecting the best-case performance
- p99 latency is the average latency measured over 99% of uptime, excluding outlier periods during incidents
Correct answer: p99 is the 99th percentile latency — 99% of requests complete faster than this value; it captures the experience of the worst-served 1% of users, which average latency hides by diluting outliers in the mean
Percentile latency (p95, p99, p999) captures tail latency experienced by real users. An average latency of 50ms can hide that 1% of users experience 5-second responses — p99 makes this visible and actionable.
The problem with average latency as an SLI: it is a mathematical mean that dilutes extreme values. Example: 99 requests complete in 10ms, 1 request takes 10,000ms (10s). Average = (99×10 + 10,000) / 100 = 109.9ms — looks fast! p99 = 10,000ms — reveals the severe tail. Percentile latency captures the actual experience of specific user percentiles: p50 (median): half of users see latency at or below this value. p95: 95% of users are at or below this. p99: the slowest 1% of users (in a service handling 10,000 req/s, that's 100 users per second). p999: the slowest 0.1% (often reveals the absolute worst cases). For SLOs: 'p99 latency under 500ms' is a meaningful commitment about the experience of nearly all users. 'Average latency under 100ms' could be met even if 5% of users experience 2-second responses. Google's SRE model recommends SLIs based on proportional metrics (fraction of requests meeting the threshold), which aligns with percentile monitoring: 'proportion of requests served under 500ms' is equivalent to monitoring that the p99 is under 500ms.
Question 4: What is 'autoscaling,' and what metric is MOST appropriate to trigger autoscaling for a web API service?
- Autoscaling adjusts the number of running instances based on demand; for a web API, scaling based on requests per second (RPS) or concurrent connections is more appropriate than CPU utilization alone, as some APIs are I/O-bound rather than CPU-bound (Correct answer)
- Autoscaling should always use CPU utilization as the trigger because it is the most accurate proxy for user-facing load across all service types
- Autoscaling based on memory utilization is most appropriate because memory is the limiting resource for most web applications
- Autoscaling is only appropriate for stateless services; stateful services must be manually scaled to avoid data consistency issues
Correct answer: Autoscaling adjusts the number of running instances based on demand; for a web API, scaling based on requests per second (RPS) or concurrent connections is more appropriate than CPU utilization alone, as some APIs are I/O-bound rather than CPU-bound
CPU-only autoscaling fails for I/O-bound services where CPU stays low even under heavy load because threads are waiting on database or network I/O. Request rate or concurrent connection metrics better represent actual service demand.
Autoscaling trigger selection depends on the bottleneck resource for the specific service: CPU-bound services (image processing, ML inference, encryption): CPU utilization is an appropriate autoscaling metric. I/O-bound services (most web APIs that query databases or call downstream services): CPU stays low while threads block on I/O. RPS, concurrent connection count, or queue depth are better autoscaling triggers. Memory-bound services: memory utilization may be appropriate, but memory-based autoscaling can cause thrashing (scale up adds instances, each fills memory, triggers more scaling). Custom application metrics (via Prometheus KEDA or AWS Custom Metrics): often the most accurate — HTTP request queue depth directly reflects load. Kubernetes implementation: HPA with custom metrics adapter (Prometheus Adapter) can scale on RPS. KEDA (Kubernetes Event-Driven Autoscaling) can scale on Kafka consumer lag, SQS queue depth, or any external metric. Best practice: always validate your autoscaling configuration with a load test — confirm the scaler triggers before the service starts to degrade, and confirm it scales down appropriately to control costs.
Question 5: What is the 'golden signals' framework and which four metrics does it include?
- Latency, traffic (throughput), errors, and saturation — the four signals that best describe the health of any service from a user and capacity perspective (Correct answer)
- CPU, memory, disk I/O, and network bandwidth — the four system-level metrics that determine service performance
- Availability, reliability, performance, and security — the four dimensions of service quality per the NIST framework
- Mean time to detect, mean time to respond, mean time to recover, and mean time between failures — the four incident response quality metrics
Correct answer: Latency, traffic (throughput), errors, and saturation — the four signals that best describe the health of any service from a user and capacity perspective
Google's four golden signals (from the SRE book) are the minimum set of metrics needed to understand service health: latency (how fast), traffic (how much load), errors (how much is failing), and saturation (how full/close to limit).
The four golden signals, from Google's 'Site Reliability Engineering' book: (1) Latency: the time it takes to service a request. Distinguish between successful request latency and error latency (an error that returns immediately is less damaging than a slow error). Use percentiles (p50, p99), not averages. (2) Traffic: a measure of how much demand is being placed on the system. For web services: HTTP requests per second. For streaming: network I/O rate. For databases: queries per second. (3) Errors: the rate of requests that fail. Distinguish between explicit failures (HTTP 5xx), implicit failures (HTTP 200 but with incorrect content), and policy violations (responses above the latency SLO even if successful). (4) Saturation: how 'full' the service is. Usually measured as utilization of the binding constraint: CPU utilization for CPU-bound services, memory for memory-bound services, I/O queue depth for disk-bound services. Saturation predicts future degradation — a service at 90% utilization is near its performance cliff. The four golden signals are the recommended minimum monitoring setup for any service, from which you build more specific dashboards and SLIs.
Question 6: What is 'capacity planning' in SRE, and what data is MOST critical for generating accurate capacity forecasts?
- Capacity planning ensures service resources are provisioned to handle current and projected future load; the most critical inputs are historical traffic growth trends, SLO performance at various utilization levels from load tests, and upcoming business events that could cause traffic spikes (Correct answer)
- Capacity planning is purely a cost optimization exercise focused on minimizing infrastructure spend
- Capacity planning only requires knowing the current peak traffic level and doubling it for a conservative estimate
- Capacity planning is the responsibility of finance teams, not SREs — SREs only respond to capacity problems after they cause incidents
Correct answer: Capacity planning ensures service resources are provisioned to handle current and projected future load; the most critical inputs are historical traffic growth trends, SLO performance at various utilization levels from load tests, and upcoming business events that could cause traffic spikes
Accurate capacity forecasting requires historical growth trends (how fast is traffic growing?), SLO performance curves from load testing (at what utilization does latency degrade?), and business event calendars (when will traffic spikes occur and how large?).
SRE capacity planning involves: (1) Historical traffic analysis: measure week-over-week and month-over-month traffic growth rate. Extrapolate to find when current capacity will be exhausted. (2) SLO performance modeling: from load tests, know at what utilization level SLOs degrade. Typically: maintain peak traffic at ≤60-70% of capacity to preserve headroom for spikes. (3) Business event calendar: identify planned events (product launches, marketing campaigns, seasonal peaks, system migrations) and their expected traffic impact. (4) Capacity buffer: add headroom for unexpected traffic spikes (typically 20-40% above expected peak). (5) Lead time: account for the time required to provision additional capacity (cloud: minutes; bare metal: weeks to months). Capacity planning outputs: when to scale up, by how much, what the cost will be, and what the capacity ceiling is before SLOs are at risk. SRE involvement is critical because only SREs know the correlation between traffic load and SLO performance from load tests — finance teams cannot calculate this without that engineering input.
SRE Performance Testing and Load Management 1
What is the difference between a 'load test' and a 'stress test,' and when should each be used?