Kafka Architecture & Internals Flashcards

What is a Kafka 'offset'?

Answer: A sequential integer that uniquely identifies each message's position within a partition

An offset is a monotonically increasing integer assigned to each message within a partition, used by consumers to track which messages have been read.

What does 'exactly-once semantics' (EOS) in Kafka require?

Answer: Idempotent producers plus Kafka transactions to ensure each message is processed and committed once

EOS requires idempotent producers (no duplicates) combined with Kafka's transactional API so that reads, processing, and writes are committed atomically.

What is a Kafka 'transaction'?

Answer: An atomic operation that writes to multiple topics/partitions and commits consumer offsets as a single all-or-nothing unit

Kafka transactions allow producers to send messages to multiple partitions and commit consumer offsets atomically, ensuring all-or-nothing semantics.

What is the role of the Kafka broker's 'log cleaner' thread?

Answer: Performs log compaction by removing older duplicate-key records from compacted topics

The log cleaner is a background thread that runs compaction on topics with cleanup.policy=compact, merging segments and removing superseded key records.

What is 'zero-copy' in the context of Kafka's I/O performance?

Answer: Data is transferred from disk to network buffer directly in the OS kernel without copying through user space

Kafka uses the OS sendfile() system call to transfer data from the page cache to the network socket entirely in kernel space, dramatically reducing CPU and memory overhead.

What is the 'page cache' and why is it important for Kafka performance?

Answer: The OS-level disk cache that Kafka relies on to serve repeated reads without physical disk I/O

Kafka intentionally avoids maintaining its own in-process cache and instead relies on the OS page cache, allowing hot data to be served from memory by the kernel.