Kafka Architecture & Internals Flashcards

What is the Kafka 'high-water mark' (HWM)?

Answer: The highest offset that has been replicated to all in-sync replicas and is safe to expose to consumers

The high-water mark is the offset up to which all ISR replicas have acknowledged data; consumers can only read up to this offset to ensure consistency.

What does the Kafka log-end offset (LEO) represent?

Answer: The next offset to be written, i.e., the offset of the last record + 1 in a replica's log

The log-end offset is the offset of the next message to be appended; it represents the tip of a replica's local log and may exceed the high-water mark on followers.

How does a follower replica know it is out of sync with the leader?

Answer: When its LEO lags behind the leader's LEO for more than replica.lag.time.max.ms

A replica is removed from the ISR when it hasn't caught up with the leader's log within replica.lag.time.max.ms, indicating it is behind.

What is a 'preferred replica election' in Kafka?

Answer: A process that restores partition leadership to the originally assigned (preferred) broker after a failover

After a broker failure and recovery, preferred replica election moves leadership back to the originally assigned broker to achieve balanced load distribution.

What is the function of the Kafka `__transaction_state` internal topic?

Answer: Stores the state of ongoing and completed transactions managed by transaction coordinators

__transaction_state is Kafka's internal topic for persisting transaction status so that transaction coordinators can recover after a failure.

Why does Kafka store messages on disk sequentially?

Answer: Sequential disk writes are extremely fast and leverage OS prefetching, making them comparable to in-memory speeds

Sequential I/O on modern hardware is orders of magnitude faster than random I/O; Kafka's append-only log design exploits this along with OS read-ahead prefetching.