SRE Kubernetes and Container Orchestration 1 — Questions and Answers
Question 1: What is the purpose of a Kubernetes 'liveness probe,' and how does it differ from a 'readiness probe'?
- A liveness probe restarts a container that has deadlocked; a readiness probe removes a container from load balancer rotation when it cannot serve traffic (Correct answer)
- A liveness probe checks if a container is in the load balancer pool; a readiness probe checks if the container process is running
- A liveness probe validates Kubernetes node health; a readiness probe validates pod health
- Both probes perform identical checks; the difference is only in which team configures them
Correct answer: A liveness probe restarts a container that has deadlocked; a readiness probe removes a container from load balancer rotation when it cannot serve traffic
Liveness probes detect deadlocked or corrupted container states and trigger a restart. Readiness probes detect temporary unavailability (e.g., loading config) and temporarily remove the pod from service endpoints without restarting it.
Kubernetes has three types of probes: Liveness probes determine if a container is still alive. If the probe fails, kubelet kills the container and restarts it (subject to the restart policy). Use case: a Java application that has entered a deadlock state where threads are blocked but the process is still running. Readiness probes determine if a container is ready to receive traffic. If the probe fails, the pod is removed from the Service's Endpoints (no new requests are routed to it), but the container is NOT restarted. Use case: a server that is temporarily loading a large ML model and should not receive traffic until loading is complete. Startup probes are a third type, for slow-starting containers, that delay liveness/readiness checks until the application has started. Configuring these probes correctly is critical — aggressive liveness probes that kill healthy pods are a common source of reliability incidents.
Question 2: A Kubernetes Deployment has 3 replicas. During a rolling update, what is the effect of setting maxSurge=1 and maxUnavailable=0?
- At most 4 pods run simultaneously during the update (3 original + 1 new), and no pods are terminated until a new one is healthy — ensuring continuous availability (Correct answer)
- All 3 original pods are terminated and then 3 new pods are created, resulting in brief downtime
- The update creates 1 new pod and terminates 3 old pods simultaneously, reducing capacity to 1 during the transition
- maxSurge=1 and maxUnavailable=0 are incompatible settings that will cause the deployment to fail
Correct answer: At most 4 pods run simultaneously during the update (3 original + 1 new), and no pods are terminated until a new one is healthy — ensuring continuous availability
maxSurge=1 allows 1 extra pod above the desired count (4 total), maxUnavailable=0 ensures no reduction in available pods. New pods must become ready before old ones are terminated, guaranteeing zero downtime during the rollout.
Rolling update parameters control the deployment rollout strategy: maxSurge specifies the maximum number of pods above the desired replica count that can exist during the update (can be an absolute number or a percentage). maxUnavailable specifies the maximum number of pods that can be unavailable during the update. With desired=3, maxSurge=1, maxUnavailable=0: Step 1: Create 1 new pod (total: 4 pods, 3 old + 1 new). Step 2: Wait for the new pod to pass readiness checks. Step 3: Terminate 1 old pod (total: 3 pods, 2 old + 1 new). Step 4: Repeat. This rolling process ensures the service always has at least 3 ready pods. For maxUnavailable=1, maxSurge=0: old pods are terminated before new ones are created — faster but causes temporary reduction in capacity.
Question 3: What is a Kubernetes PodDisruptionBudget (PDB) and when is it essential?
- A PDB specifies the minimum number or percentage of pods that must remain available during voluntary disruptions like node drains, cluster upgrades, or maintenance (Correct answer)
- A PDB limits the CPU and memory resources a pod can consume to prevent resource contention
- A PDB defines the maximum number of pods that can be scheduled on a single node
- A PDB controls the rate at which pods are created during a deployment rollout
Correct answer: A PDB specifies the minimum number or percentage of pods that must remain available during voluntary disruptions like node drains, cluster upgrades, or maintenance
A PodDisruptionBudget ensures that voluntary disruptions (like kubectl drain during a node upgrade) do not bring down too many pods simultaneously, preventing inadvertent downtime during maintenance operations.
PodDisruptionBudgets protect applications from simultaneous pod evictions during voluntary disruptions. Voluntary disruptions include: node drains (moving pods off a node for maintenance), cluster version upgrades (where nodes are upgraded rolling), and autoscaler-driven scale-down events. A PDB with minAvailable=2 on a 3-replica deployment means that a cluster upgrade can only evict 1 pod at a time — the second pod will not be evicted until the first is rescheduled and running. Without a PDB, a node drain could evict all pods on that node simultaneously, causing a service outage during a routine maintenance window. PDBs are essential for: stateful applications, services with small replica counts, and any service where losing multiple instances simultaneously causes an outage. Note: PDBs only apply to voluntary disruptions; node failures (involuntary disruptions) bypass PDBs.
Question 4: In Kubernetes, what is the difference between a ConfigMap and a Secret, and what are the security implications?
- ConfigMaps store non-sensitive configuration data as plain text; Secrets store sensitive data base64-encoded (not encrypted by default), and should be protected with RBAC, encryption at rest, and preferably external secret managers (Correct answer)
- ConfigMaps are cluster-scoped; Secrets are namespace-scoped, providing stronger isolation for sensitive data
- Secrets are automatically encrypted by Kubernetes before storage; ConfigMaps are stored in plain text in etcd
- ConfigMaps can be mounted as volumes; Secrets can only be injected as environment variables to prevent disk persistence
Correct answer: ConfigMaps store non-sensitive configuration data as plain text; Secrets store sensitive data base64-encoded (not encrypted by default), and should be protected with RBAC, encryption at rest, and preferably external secret managers
ConfigMaps are for non-sensitive configuration; Secrets are for sensitive data but are only base64-encoded (not encrypted) by default in etcd. Proper secret security requires encryption at rest, RBAC restrictions, and ideally external secret management systems.
A critical Kubernetes security misunderstanding: Kubernetes Secrets are NOT encrypted by default. They are stored base64-encoded in etcd, which means anyone with etcd read access can trivially decode them. Secure secret management in Kubernetes requires: (1) etcd encryption at rest (configured via EncryptionConfiguration). (2) RBAC policies that limit which service accounts and users can read Secrets. (3) Avoid mounting secrets as environment variables (more likely to be logged accidentally) — prefer volume mounts with strict permissions. (4) External secret managers (HashiCorp Vault, AWS Secrets Manager, GCP Secret Manager) with operators like External Secrets Operator, which store the actual secret values outside Kubernetes and inject them at pod startup. (5) Regular rotation of secrets. Both ConfigMaps and Secrets can be mounted as volumes or injected as environment variables.
Question 5: What does Kubernetes horizontal pod autoscaling (HPA) do, and what metric is it MOST commonly configured to use?
- HPA automatically adjusts the number of pod replicas based on observed metrics (most commonly CPU utilization), scaling out when demand increases and in when it decreases (Correct answer)
- HPA adjusts the CPU and memory limits of running pods based on their actual usage without changing the replica count
- HPA distributes pods across nodes to maximize hardware utilization based on CPU load
- HPA monitors pod health and replaces unhealthy pods with new ones during high-load periods
Correct answer: HPA automatically adjusts the number of pod replicas based on observed metrics (most commonly CPU utilization), scaling out when demand increases and in when it decreases
HPA watches metrics (default: CPU utilization as a percentage of the CPU request) and scales the number of replicas up or down to maintain the target metric value. It is the primary autoscaling mechanism for request-driven workloads.
Kubernetes Horizontal Pod Autoscaler (HPA) works by: (1) Periodically querying metrics (from metrics-server for CPU/memory, or from custom metrics API for application-specific metrics like request rate or queue depth). (2) Computing the desired replica count: current replicas × (current metric value / target metric value). (3) Scaling up immediately when demand exceeds the target, scaling down more slowly (with a cooldown window) to prevent thrashing. Common configurations: CPU-based (most common, suitable for CPU-bound services), custom metrics (HTTP request rate via Prometheus adapter, very common for web services), external metrics (queue depth from Kafka or SQS, common for worker services). SRE considerations: set minReplicas to ensure base availability, set maxReplicas to cap cost, and ensure the target metric actually correlates with user impact.
Question 6: A microservice running in Kubernetes is experiencing intermittent OOMKilled events. What is the MOST appropriate first response?
- Analyze memory usage patterns using profiling tools to identify memory leaks or excessive allocation, then set appropriate memory limits and requests based on observed usage (Correct answer)
- Increase the memory limit to 10× the current value to prevent future OOMKilled events
- Remove all memory limits so Kubernetes does not kill the pod when memory spikes
- Scale the number of replicas to distribute memory load across more pods
Correct answer: Analyze memory usage patterns using profiling tools to identify memory leaks or excessive allocation, then set appropriate memory limits and requests based on observed usage
OOMKilled events indicate the container exceeded its memory limit. The correct approach is to investigate whether the limit is too low for legitimate usage or whether there is a memory leak, then set limits based on profiled actual usage.
OOMKilled (Out Of Memory Killed) occurs when a container's memory usage exceeds its memory limit, causing Linux to kill the process. Proper diagnosis: (1) Check if memory usage is growing continuously (memory leak) or spikes during specific operations (correct behavior hitting a tight limit). (2) Use profiling tools (pprof for Go, heap dumps for Java, memory_profiler for Python) to identify allocation sources. (3) Monitor memory usage over time with metrics. (4) Set the memory limit at the 99th percentile of observed usage plus a safe headroom (typically 20–30%). Arbitrarily increasing limits by 10× (option B) wastes resources and may mask a leak. Removing limits entirely (option C) allows a buggy pod to consume all node memory, causing other pods on the same node to be killed. Scaling replicas (option D) distributes requests but each replica will still hit the same per-pod limit.
SRE Kubernetes and Container Orchestration 1
What is the purpose of a Kubernetes 'liveness probe,' and how does it differ from a 'readiness probe'?