Data Engineering Distributed Data Processing 3 — Questions and Answers
Question 1: Which delivery guarantee ensures each message is processed once with no duplicates or loss?
- Exactly-once (Correct answer)
- At-least-once
- At-most-once
- Best-effort
Correct answer: Exactly-once
Exactly-once semantics processes every record one time, avoiding duplicates and drops.
Question 2: In Kafka, what determines the maximum parallelism of consumers in a group?
- The number of partitions (Correct answer)
- The number of brokers
- The replication factor
- The retention period
Correct answer: The number of partitions
Each partition is consumed by at most one consumer in a group, capping parallelism.
Question 3: What is the main trade-off described by the CAP theorem during a network partition?
- Choosing between consistency and availability (Correct answer)
- Choosing between speed and storage
- Choosing between SQL and NoSQL
- Choosing between batch and stream
Correct answer: Choosing between consistency and availability
Under a partition you must sacrifice either consistency or availability.
Question 4: A windowed aggregation over event-time streams typically needs which mechanism to handle late data?
- Watermarks (Correct answer)
- Replication factor
- Bloom filters
- Columnar storage
Correct answer: Watermarks
Watermarks define how long to wait for late events before finalizing a window.
Question 5: Why might increasing the number of partitions improve throughput up to a point?
- More partitions allow more parallel tasks (Correct answer)
- It reduces data size
- It removes the shuffle stage
- It guarantees ordering
Correct answer: More partitions allow more parallel tasks
More partitions enable greater parallelism, but too many add scheduling overhead.
Question 6: What is a coordinator's role in a two-phase commit protocol?
- Asking participants to prepare, then commit or abort (Correct answer)
- Storing all data permanently
- Encrypting the network
- Partitioning the input
Correct answer: Asking participants to prepare, then commit or abort
The coordinator runs prepare and commit phases to keep a distributed transaction atomic.
Question 7: Which storage format is best suited for analytical column scans across distributed engines?
- Parquet (Correct answer)
- CSV
- Plain text logs
- XML
Correct answer: Parquet
Parquet's columnar layout lets engines read only needed columns efficiently.
Data Engineering Distributed Data Processing 3
Which delivery guarantee ensures each message is processed once with no duplicates or loss?