Apache Kafka Kafka Architecture & Internals 2 — Questions and Answers
Question 1: What is the Kafka 'high-water mark' (HWM)?
- The highest offset that has been replicated to all in-sync replicas and is safe to expose to consumers (Correct answer)
- The maximum bytes a broker stores before truncating
- The last committed consumer group offset
- The maximum lag threshold before alerting
Correct answer: The highest offset that has been replicated to all in-sync replicas and is safe to expose to consumers
The high-water mark is the offset up to which all ISR replicas have acknowledged data; consumers can only read up to this offset to ensure consistency.
Question 2: What does the Kafka log-end offset (LEO) represent?
- The next offset to be written, i.e., the offset of the last record + 1 in a replica's log (Correct answer)
- The last offset consumed by all consumer groups
- The high-water mark for the partition
- The first offset in the current active segment
Correct answer: The next offset to be written, i.e., the offset of the last record + 1 in a replica's log
The log-end offset is the offset of the next message to be appended; it represents the tip of a replica's local log and may exceed the high-water mark on followers.
Question 3: How does a follower replica know it is out of sync with the leader?
- When its LEO lags behind the leader's LEO for more than replica.lag.time.max.ms (Correct answer)
- When it receives a different partition ID from ZooKeeper
- When it fails to compress messages correctly
- When it misses an ACK within acks.timeout.ms
Correct answer: When its LEO lags behind the leader's LEO for more than replica.lag.time.max.ms
A replica is removed from the ISR when it hasn't caught up with the leader's log within replica.lag.time.max.ms, indicating it is behind.
Question 4: What is a 'preferred replica election' in Kafka?
- A process that restores partition leadership to the originally assigned (preferred) broker after a failover (Correct answer)
- An election that selects the fastest replica as leader
- A manual ACL grant for replica access
- Automatic rebalancing of topic leaders across brokers
Correct answer: A process that restores partition leadership to the originally assigned (preferred) broker after a failover
After a broker failure and recovery, preferred replica election moves leadership back to the originally assigned broker to achieve balanced load distribution.
Question 5: What is the function of the Kafka `__transaction_state` internal topic?
- Stores the state of ongoing and completed transactions managed by transaction coordinators (Correct answer)
- Logs all broker configuration changes
- Tracks consumer group rebalance events
- Stores compacted key offsets for cleanup
Correct answer: Stores the state of ongoing and completed transactions managed by transaction coordinators
__transaction_state is Kafka's internal topic for persisting transaction status so that transaction coordinators can recover after a failure.
Question 6: Why does Kafka store messages on disk sequentially?
- Sequential disk writes are extremely fast and leverage OS prefetching, making them comparable to in-memory speeds (Correct answer)
- Disk is cheaper than RAM
- It avoids the need for a file index
- Sequential storage enables built-in encryption
Correct answer: Sequential disk writes are extremely fast and leverage OS prefetching, making them comparable to in-memory speeds
Sequential I/O on modern hardware is orders of magnitude faster than random I/O; Kafka's append-only log design exploits this along with OS read-ahead prefetching.
Apache Kafka Kafka Architecture & Internals 2
What is the Kafka 'high-water mark' (HWM)?