SRE Kubernetes and Container Orchestration 2 — Questions and Answers
Question 1: What is the purpose of Kubernetes resource requests and limits, and what happens when a container exceeds its CPU limit?
- Requests are used for scheduling decisions (the minimum guaranteed resources); limits cap usage. A container that exceeds its CPU limit is throttled (not killed), while exceeding memory limits causes OOMKill (Correct answer)
- Both requests and limits define the maximum resources a container can use; exceeding either causes the pod to be evicted
- Requests define the initial allocation; limits define the maximum. Exceeding CPU limits kills the container, just like exceeding memory limits
- Requests and limits are identical unless the QoS class is set to Burstable, in which case limits can exceed requests
Correct answer: Requests are used for scheduling decisions (the minimum guaranteed resources); limits cap usage. A container that exceeds its CPU limit is throttled (not killed), while exceeding memory limits causes OOMKill
CPU limits cause throttling (reduced CPU time) not termination. Memory limits cause OOMKill. Requests inform the scheduler about the minimum resources needed for a pod to be placed on a node.
Kubernetes resource management: Requests represent the guaranteed minimum resources a container needs and are used by the scheduler to determine node placement (a node must have at least this much free capacity). Limits represent the maximum a container can use. CPU behavior at limit: Linux cgroups throttle the container's CPU usage — it gets less CPU time but continues running. This causes latency increases (the application runs more slowly) but not crashes. This is why high CPU throttling can cause latency SLO violations without any pod restarts visible in monitoring. Memory behavior at limit: Linux cgroups use OOM (Out of Memory) killing to enforce memory limits. When a container exceeds its memory limit, the kernel kills the process. Kubernetes detects the OOMKill and may restart the container (depending on the RestartPolicy). Understanding this difference is critical for diagnosing latency vs. availability issues.
Question 2: What is a Kubernetes StatefulSet, and when should it be used instead of a Deployment?
- A StatefulSet provides stable network identifiers and persistent storage for each pod, making it appropriate for stateful applications like databases that require ordered deployment and stable hostnames (Correct answer)
- A StatefulSet is identical to a Deployment but adds automatic backup functionality for persistent volumes
- A StatefulSet prevents pods from being rescheduled to different nodes, ensuring they always run on the same hardware
- A StatefulSet should always be used instead of Deployments because it provides stronger consistency guarantees
Correct answer: A StatefulSet provides stable network identifiers and persistent storage for each pod, making it appropriate for stateful applications like databases that require ordered deployment and stable hostnames
StatefulSets provide stable, unique pod identifiers (pod-0, pod-1), stable DNS names, and ordered rolling updates — critical for databases and clustered applications that rely on stable member identities.
StatefulSets differ from Deployments in several key ways: (1) Stable pod identity: pods are numbered sequentially (mysql-0, mysql-1, mysql-2) and retain their identity on rescheduling. (2) Stable network identity: each pod gets a stable DNS hostname (mysql-0.mysql.default.svc.cluster.local) that persists across pod restarts. (3) Ordered deployment: pods are created in order (0 before 1 before 2) and scaled down in reverse order. (4) Persistent storage: each pod gets its own PersistentVolumeClaim that is not deleted when the pod is deleted. Use StatefulSets for: databases (MySQL, PostgreSQL, Cassandra, MongoDB), distributed caches (Redis Sentinel), message queues (Kafka, RabbitMQ), and any clustered application where members need to discover each other by name. Deployments are appropriate for stateless, interchangeable replicas where any instance can handle any request.
Question 3: What is a Kubernetes NetworkPolicy, and what is the default behavior if no NetworkPolicy exists in a namespace?
- NetworkPolicies are firewall rules for pods; by default (no NetworkPolicy), all pods can communicate freely with all other pods in the cluster — the default is allow-all (Correct answer)
- NetworkPolicies control which nodes pods can be scheduled on based on network topology
- NetworkPolicies are the default configuration — if none exist, all pod-to-pod communication is blocked
- NetworkPolicies only apply to traffic entering the cluster from external sources, not pod-to-pod traffic
Correct answer: NetworkPolicies are firewall rules for pods; by default (no NetworkPolicy), all pods can communicate freely with all other pods in the cluster — the default is allow-all
By default, Kubernetes allows all pod-to-pod communication across all namespaces (open network). NetworkPolicies selectively restrict this by defining ingress/egress rules — but require a CNI plugin that supports them (e.g., Calico, Cilium) to take effect.
Kubernetes' default network model is flat and open: every pod can reach every other pod by IP address, regardless of namespace. This is convenient for getting started but violates the principle of least privilege for production environments. NetworkPolicies work by: (1) Selecting which pods they apply to (using label selectors). (2) Defining allowed ingress sources (which pods/namespaces/CIDR blocks can send traffic to the selected pods). (3) Defining allowed egress destinations. Important caveats: NetworkPolicies are additive — having any NetworkPolicy on a pod makes all non-whitelisted traffic denied. NetworkPolicies require a CNI plugin that supports them (kube-proxy alone doesn't enforce them). Best practices: implement a default-deny policy for both ingress and egress in each namespace, then explicitly allow required communication paths. This is called a zero-trust network model.
Question 4: What is the Kubernetes Operator pattern, and what problem does it solve?
- Operators encode the knowledge of running a complex stateful application (e.g., a database) as code, automating tasks like provisioning, scaling, backups, and failure recovery that would otherwise require manual expertise (Correct answer)
- Operators are Kubernetes cluster administrators who have elevated RBAC permissions to manage all resources
- Operators are load-balancing controllers that distribute traffic across pods using advanced algorithms
- Operators are monitoring agents that collect metrics from pods and send them to external observability systems
Correct answer: Operators encode the knowledge of running a complex stateful application (e.g., a database) as code, automating tasks like provisioning, scaling, backups, and failure recovery that would otherwise require manual expertise
The Operator pattern extends the Kubernetes API with custom resources and controllers that automate the lifecycle management of complex applications, encoding the operational expertise of a human administrator into software.
The Operator pattern, introduced by CoreOS in 2016, extends the Kubernetes control loop model to complex stateful applications. A standard Kubernetes controller (e.g., Deployment controller) only knows how to manage generic pods. An Operator adds: (1) Custom Resource Definitions (CRDs) — new Kubernetes object types (e.g., PostgreSQLCluster). (2) A custom controller that watches these CRDs and takes action to reconcile the desired state with actual state. (3) Encoded operational knowledge: how to initialize a cluster, add/remove members, perform rolling upgrades, take/restore backups, handle failover, and respond to specific failure modes. Examples: the PostgreSQL Operator (Zalando), Prometheus Operator, Elasticsearch (ECK) Operator, Cassandra Operator. Operators dramatically reduce toil for running complex stateful workloads in Kubernetes by automating what would otherwise require deep expertise and manual intervention.
Question 5: A Kubernetes node is marked as 'NotReady.' What is the MOST likely impact on pods running on that node?
- After the node remains NotReady beyond the pod eviction timeout (default 5 minutes), the pods are evicted and rescheduled on healthy nodes, subject to pod disruption budgets (Correct answer)
- Pods on the NotReady node immediately terminate and are recreated on other nodes
- Pods on the NotReady node continue running until their next restart cycle, which may be hours later
- Kubernetes automatically increases the resource limits of pods on other nodes to compensate for the lost capacity
Correct answer: After the node remains NotReady beyond the pod eviction timeout (default 5 minutes), the pods are evicted and rescheduled on healthy nodes, subject to pod disruption budgets
Kubernetes does not immediately evict pods from a NotReady node — it waits for the configured toleration seconds (default: 5 minutes for node.kubernetes.io/not-ready) before evicting and rescheduling pods, providing a buffer for transient node issues.
When a node becomes NotReady, the node lifecycle controller follows this sequence: (1) The node's status is updated to NotReady by the node controller, which sets the node.kubernetes.io/not-ready taint. (2) Pods have a default toleration for this taint with tolerationSeconds=300 (5 minutes). (3) After 5 minutes of continued NotReady status, the pods are considered failed and the scheduler evicts them and creates replacement pods on healthy nodes. This delay is intentional — it prevents mass pod churn during transient issues like a brief kubelet restart or network blip. Pods that are part of Deployments or StatefulSets are automatically recreated; standalone pods (no controller) are not recreated. PodDisruptionBudgets are checked during the rescheduling process. The eviction timeout can be tuned by modifying the pod's tolerations.
Question 6: What is the role of etcd in a Kubernetes cluster, and why is it critical to its reliability?
- etcd is the distributed key-value store that persists all cluster state (pod specs, ConfigMaps, Secrets, RBAC policies, etc.); if etcd is unavailable, the API server cannot function and the cluster cannot be managed (Correct answer)
- etcd is the container runtime that manages the lifecycle of containers on each worker node
- etcd is the network overlay plugin that handles pod-to-pod communication across nodes
- etcd is an optional caching layer that improves API server read performance by storing frequently accessed resources
Correct answer: etcd is the distributed key-value store that persists all cluster state (pod specs, ConfigMaps, Secrets, RBAC policies, etc.); if etcd is unavailable, the API server cannot function and the cluster cannot be managed
etcd is Kubernetes' single source of truth — every object definition, configuration, and status is persisted in etcd. Loss of etcd quorum means no cluster state changes can be made, and restoring from backup may be the only recovery option.
etcd is a distributed, strongly consistent key-value store that serves as Kubernetes' backing store for all cluster data. Every kubectl apply, pod creation, ConfigMap update, and RBAC policy change is ultimately stored in etcd. etcd runs as a distributed cluster (typically 3 or 5 nodes in production) for high availability, using the Raft consensus algorithm to ensure consistency. Reliability implications: (1) etcd must maintain quorum (majority of nodes available) to function — a 3-node cluster can tolerate 1 node failure. (2) etcd is the critical backup target — without a recent etcd backup, a cluster failure means losing all workload definitions. (3) etcd performance directly affects API server latency — slow etcd causes slow kubectl responses and delayed controller reactions. (4) etcd should be on fast SSD storage with low latency — disk I/O is etcd's primary performance bottleneck. Production Kubernetes requires dedicated etcd infrastructure with regular backups and tested restore procedures.
SRE Kubernetes and Container Orchestration 2
What is the purpose of Kubernetes resource requests and limits, and what happens when a container exceeds its CPU limit?