Data Engineering Data Ingestion Patterns 2

Question 1

A retail company needs to load order events into a warehouse with at most a few minutes of delay, but does not require sub-second latency. Which ingestion pattern best fits?

Accepted Answer

Micro-batch ingestion

Answer

Micro-batch ingestion processes small batches every few minutes, balancing freshness with throughput efficiency.

Question 2

In Change Data Capture (CDC), which approach reads the database transaction log to detect changes?

Accepted Answer

Log-based CDC

Answer

Log-based CDC reads the database write-ahead/transaction log, capturing changes with minimal load on the source.

Question 3

What is the primary advantage of a pull-based ingestion model over a push-based one?

Accepted Answer

The consumer controls the rate and timing of data retrieval

Answer

In pull-based ingestion the consumer polls the source, controlling retrieval rate and avoiding being overwhelmed.

Question 4

Which scenario most justifies using a streaming ingestion pattern instead of batch?

Accepted Answer

Fraud detection that must react within seconds

Answer

Fraud detection requires immediate reaction, making low-latency streaming ingestion appropriate.

Question 5

When ingesting from a REST API that paginates results, what mechanism prevents missing or duplicating records across pages?

Accepted Answer

A stable cursor or continuation token

Answer

A stable cursor/continuation token gives a consistent position so pages neither overlap nor skip records.

Data Engineering Practice Test

Data Engineering Practice Test

Data Engineering Data Ingestion Patterns 2