Apache Kafka Kafka Architecture & Internals 1 — Questions and Answers
Question 1: What is a Kafka 'offset'?
- A sequential integer that uniquely identifies each message's position within a partition (Correct answer)
- The number of bytes from the start of a log file
- A consumer group's processing delay metric
- A broker-assigned message UUID
Correct answer: A sequential integer that uniquely identifies each message's position within a partition
An offset is a monotonically increasing integer assigned to each message within a partition, used by consumers to track which messages have been read.
Question 2: What does 'exactly-once semantics' (EOS) in Kafka require?
- Idempotent producers plus Kafka transactions to ensure each message is processed and committed once (Correct answer)
- Only acks=1 on the producer
- Setting max.in.flight.requests.per.connection=1
- Enabling log compaction on the topic
Correct answer: Idempotent producers plus Kafka transactions to ensure each message is processed and committed once
EOS requires idempotent producers (no duplicates) combined with Kafka's transactional API so that reads, processing, and writes are committed atomically.
Question 3: What is a Kafka 'transaction'?
- An atomic operation that writes to multiple topics/partitions and commits consumer offsets as a single all-or-nothing unit (Correct answer)
- A scheduled batch job in Kafka
- A distributed transaction spanning external databases
- A ZooKeeper lock acquired during produce
Correct answer: An atomic operation that writes to multiple topics/partitions and commits consumer offsets as a single all-or-nothing unit
Kafka transactions allow producers to send messages to multiple partitions and commit consumer offsets atomically, ensuring all-or-nothing semantics.
Question 4: What is the role of the Kafka broker's 'log cleaner' thread?
- Performs log compaction by removing older duplicate-key records from compacted topics (Correct answer)
- Deletes expired messages based on retention.ms
- Compresses log segment files
- Truncates follower logs during leader election
Correct answer: Performs log compaction by removing older duplicate-key records from compacted topics
The log cleaner is a background thread that runs compaction on topics with cleanup.policy=compact, merging segments and removing superseded key records.
Question 5: What is 'zero-copy' in the context of Kafka's I/O performance?
- Data is transferred from disk to network buffer directly in the OS kernel without copying through user space (Correct answer)
- Messages are stored without serialization overhead
- No replicas means no copy overhead
- The producer reuses message objects without copying
Correct answer: Data is transferred from disk to network buffer directly in the OS kernel without copying through user space
Kafka uses the OS sendfile() system call to transfer data from the page cache to the network socket entirely in kernel space, dramatically reducing CPU and memory overhead.
Question 6: What is the 'page cache' and why is it important for Kafka performance?
- The OS-level disk cache that Kafka relies on to serve repeated reads without physical disk I/O (Correct answer)
- An in-memory buffer inside each broker JVM for recent messages
- A Kafka-specific caching layer in front of ZooKeeper
- A client-side cache for recently consumed offsets
Correct answer: The OS-level disk cache that Kafka relies on to serve repeated reads without physical disk I/O
Kafka intentionally avoids maintaining its own in-process cache and instead relies on the OS page cache, allowing hot data to be served from memory by the kernel.
Apache Kafka Kafka Architecture & Internals 1
What is a Kafka 'offset'?