LFC Structured Streaming Pipelines 2

Question 1

In Databricks Structured Streaming, what does the `checkpointLocation` option store?

Accepted Answer

The progress of the stream and state information for recovery

Answer

The checkpoint location persists stream progress and operator state so the pipeline can recover exactly where it left off after a failure.

Question 2

Which output mode should you choose when your Structured Streaming query aggregates data and you only want to emit rows whose aggregate values have changed?

Accepted Answer

Update

Answer

Update mode emits only the rows that changed since the last trigger, making it efficient for aggregations that produce incremental changes.

Question 3

What is the primary purpose of a watermark in Structured Streaming?

Accepted Answer

To limit how late event-time data can arrive before being dropped from stateful aggregations

Answer

A watermark tells Spark the maximum acceptable lateness for late-arriving events; data older than the watermark threshold is dropped from stateful operations.

Question 4

When using Auto Loader in Databricks, what happens when new files arrive in the monitored cloud storage path?

Accepted Answer

Auto Loader uses file notification or directory listing to incrementally ingest only new files

Answer

Auto Loader efficiently tracks new files using either cloud file notification services or incremental directory listing, avoiding full rescans.

Question 5

In a Structured Streaming pipeline writing to Delta Lake, which feature prevents duplicate records when the same micro-batch is reprocessed after a failure?

Accepted Answer

Idempotent writes enabled by transactional checkpointing and Delta's ACID guarantees

Answer

Delta Lake's ACID transactions combined with Structured Streaming's checkpointing ensure each micro-batch is committed exactly once, preventing duplicates on retry.

(LFC) Lakehouse Fundamentals Certification Practice Test

(LFC) Lakehouse Fundamentals Certification Practice Test

LFC Structured Streaming Pipelines 2