Data Engineering Data Ingestion Patterns 3 — Questions and Answers
Question 1: Which pattern decouples data producers from consumers and buffers bursts of incoming events?
- A message queue or streaming log (Correct answer)
- A direct database write from the producer
- A shared spreadsheet
- A synchronous API call
Correct answer: A message queue or streaming log
A message queue or streaming log (e.g., Kafka) buffers events and decouples producers from consumers, absorbing bursts.
Question 2: In an incremental batch ingestion using a high-water mark, what value is typically stored between runs?
- The maximum updated_at/timestamp processed so far (Correct answer)
- The total number of database tables
- The source server hostname
- The average row size
Correct answer: The maximum updated_at/timestamp processed so far
A high-water mark stores the latest processed timestamp or ID so the next run only fetches newer records.
Question 3: What is the main risk of trigger-based CDC compared to log-based CDC?
- Added write overhead and latency on the source database (Correct answer)
- Inability to detect deletes
- Requiring a data warehouse
- Producing only batch output
Correct answer: Added write overhead and latency on the source database
Trigger-based CDC fires triggers on every change, adding write overhead and latency to source transactions.
Question 4: When a streaming consumer cannot keep up with the producer, which technique signals the producer to slow down?
- Backpressure (Correct answer)
- Sharding
- Denormalization
- Compaction
Correct answer: Backpressure
Backpressure is the mechanism that throttles upstream producers when downstream consumers fall behind.
Question 5: Which delivery guarantee may result in duplicate records but never lost ones?
- At-least-once (Correct answer)
- At-most-once
- Exactly-once
- Best-effort with no retries
Correct answer: At-least-once
At-least-once delivery retries until acknowledged, so records are never lost but may be delivered more than once.
Question 6: A full-load ingestion is most appropriate when:
- The source table is small or lacks a reliable change-tracking column (Correct answer)
- The table has billions of rows updated constantly
- Sub-second latency is required
- The source provides a transaction log
Correct answer: The source table is small or lacks a reliable change-tracking column
Full loads suit small tables or sources without a dependable change indicator, since reloading everything is simplest.
Question 7: What is a 'dead-letter queue' used for in ingestion pipelines?
- Storing messages that repeatedly fail processing for later inspection (Correct answer)
- Caching the most frequently accessed records
- Compressing old partitions
- Load balancing across consumers
Correct answer: Storing messages that repeatedly fail processing for later inspection
A dead-letter queue captures messages that cannot be processed successfully so they can be inspected and reprocessed.
Data Engineering Data Ingestion Patterns 3
Which pattern decouples data producers from consumers and buffers bursts of incoming events?