LFC Structured Streaming Pipelines 2 — Questions and Answers
Question 1: In Databricks Structured Streaming, what does the `checkpointLocation` option store?
- Raw input data for reprocessing
- The progress of the stream and state information for recovery (Correct answer)
- Output schema definitions
- Spark configuration overrides
Correct answer: The progress of the stream and state information for recovery
The checkpoint location persists stream progress and operator state so the pipeline can recover exactly where it left off after a failure.
Question 2: Which output mode should you choose when your Structured Streaming query aggregates data and you only want to emit rows whose aggregate values have changed?
- Append
- Complete
- Update (Correct answer)
- Overwrite
Correct answer: Update
Update mode emits only the rows that changed since the last trigger, making it efficient for aggregations that produce incremental changes.
Question 3: What is the primary purpose of a watermark in Structured Streaming?
- To encrypt streaming data in transit
- To limit how late event-time data can arrive before being dropped from stateful aggregations (Correct answer)
- To control the batch interval of micro-batches
- To define the schema of incoming records
Correct answer: To limit how late event-time data can arrive before being dropped from stateful aggregations
A watermark tells Spark the maximum acceptable lateness for late-arriving events; data older than the watermark threshold is dropped from stateful operations.
Question 4: When using Auto Loader in Databricks, what happens when new files arrive in the monitored cloud storage path?
- Auto Loader triggers a full directory scan on each micro-batch
- Auto Loader uses file notification or directory listing to incrementally ingest only new files (Correct answer)
- All existing files are re-ingested every trigger interval
- Auto Loader requires a manual refresh command to pick up new files
Correct answer: Auto Loader uses file notification or directory listing to incrementally ingest only new files
Auto Loader efficiently tracks new files using either cloud file notification services or incremental directory listing, avoiding full rescans.
Question 5: In a Structured Streaming pipeline writing to Delta Lake, which feature prevents duplicate records when the same micro-batch is reprocessed after a failure?
- Schema enforcement
- Idempotent writes enabled by transactional checkpointing and Delta's ACID guarantees (Correct answer)
- Auto-compaction
- Z-ordering
Correct answer: Idempotent writes enabled by transactional checkpointing and Delta's ACID guarantees
Delta Lake's ACID transactions combined with Structured Streaming's checkpointing ensure each micro-batch is committed exactly once, preventing duplicates on retry.
Question 6: What does the `trigger(availableNow=True)` option do in a Structured Streaming query?
- Runs the stream continuously with the fastest possible micro-batch interval
- Processes all available data in one or more batches then stops the stream automatically (Correct answer)
- Triggers only when data arrives via Kafka
- Sets the processing interval to 1 second
Correct answer: Processes all available data in one or more batches then stops the stream automatically
`availableNow=True` causes the stream to process all backlogged data in batches and then terminate, similar to a one-time trigger but potentially using multiple batches.
Question 7: Which Structured Streaming join type requires a watermark on at least one side to bound state size?
- Static-to-stream join
- Stream-to-stream inner join
- Stream-to-stream left outer join with time constraints
- Both B and C (Correct answer)
Correct answer: Both B and C
Stream-to-stream joins (both inner and outer with time constraints) require watermarks to bound the state maintained for matching records across the two streams.
LFC Structured Streaming Pipelines 2
In Databricks Structured Streaming, what does the `checkpointLocation` option store?