Data Engineering Data Ingestion Patterns 3

Question 1

Which pattern decouples data producers from consumers and buffers bursts of incoming events?

Accepted Answer

A message queue or streaming log

Answer

A message queue or streaming log (e.g., Kafka) buffers events and decouples producers from consumers, absorbing bursts.

Question 2

In an incremental batch ingestion using a high-water mark, what value is typically stored between runs?

Accepted Answer

The maximum updated_at/timestamp processed so far

Answer

A high-water mark stores the latest processed timestamp or ID so the next run only fetches newer records.

Question 3

What is the main risk of trigger-based CDC compared to log-based CDC?

Accepted Answer

Added write overhead and latency on the source database

Answer

Trigger-based CDC fires triggers on every change, adding write overhead and latency to source transactions.

Question 4

When a streaming consumer cannot keep up with the producer, which technique signals the producer to slow down?

Accepted Answer

Backpressure

Answer

Backpressure is the mechanism that throttles upstream producers when downstream consumers fall behind.

Question 5

Which delivery guarantee may result in duplicate records but never lost ones?

Accepted Answer

At-least-once

Answer

At-least-once delivery retries until acknowledged, so records are never lost but may be delivered more than once.

Data Engineering Practice Test