LFC Structured Streaming Pipelines 3 — Questions and Answers
Question 1: In Databricks, what is the difference between `trigger(processingTime='10 seconds')` and `trigger(once=True)` in Structured Streaming?
- `processingTime` runs one batch every 10 seconds continuously; `once` processes all available data in a single batch then stops (Correct answer)
- `processingTime` stops after 10 seconds; `once` runs indefinitely
- Both produce the same result but `once` is faster
- `processingTime` is for Kafka sources only; `once` is for file sources
Correct answer: `processingTime` runs one batch every 10 seconds continuously; `once` processes all available data in a single batch then stops
`processingTime` schedules recurring micro-batches at the specified interval, while `once` is a one-shot trigger that processes all available data and terminates.
Question 2: When a Structured Streaming pipeline reads from a Kafka topic, what does the `startingOffsets` option control?
- The Kafka consumer group rebalance strategy
- Where in the Kafka topic the stream begins reading (earliest, latest, or specific offsets) (Correct answer)
- The maximum number of records per micro-batch
- The Kafka broker connection timeout
Correct answer: Where in the Kafka topic the stream begins reading (earliest, latest, or specific offsets)
`startingOffsets` determines the initial position in the Kafka topic — `earliest` reads all available data, `latest` reads only new data, or you can specify exact offsets per partition.
Question 3: What is a key advantage of using Delta Live Tables (DLT) over manually written Structured Streaming pipelines in Databricks?
- DLT supports more data sources than manual streaming
- DLT automatically manages checkpointing, retries, schema evolution, and data quality expectations (Correct answer)
- DLT pipelines run faster because they bypass the Spark engine
- DLT eliminates the need for a Lakehouse storage layer
Correct answer: DLT automatically manages checkpointing, retries, schema evolution, and data quality expectations
DLT abstracts away operational complexity by automatically handling checkpoints, failure recovery, schema evolution, and enforcing data quality constraints through expectations.
Question 4: In Structured Streaming with stateful aggregations using event time, what happens to state for keys that are older than the watermark threshold?
- State is archived to cold storage automatically
- State for those keys is evicted and their aggregations are finalized (Correct answer)
- Those keys are re-queried from the source
- Spark raises an error requiring manual state cleanup
Correct answer: State for those keys is evicted and their aggregations are finalized
Once a key's event time falls below the current watermark, Spark finalizes its aggregation result and removes the state from memory to prevent unbounded growth.
Question 5: Which Structured Streaming output sink writes results to an in-memory table that can be queried with SQL during development and testing?
- Console sink
- Memory sink (Correct answer)
- Foreach sink
- File sink
Correct answer: Memory sink
The memory sink writes streaming results to an in-memory table registered under a given query name, making it easy to query interactively during development.
Question 6: What does `foreachBatch` enable in a Structured Streaming pipeline?
- It allows each micro-batch's output DataFrame to be processed with arbitrary batch DataFrame operations and written to multiple sinks (Correct answer)
- It parallelizes individual row processing across executors
- It replaces the trigger interval with event-driven processing
- It enforces exactly-once delivery to Kafka sinks
Correct answer: It allows each micro-batch's output DataFrame to be processed with arbitrary batch DataFrame operations and written to multiple sinks
`foreachBatch` exposes each micro-batch as a static DataFrame, enabling reuse of batch APIs, writing to multiple destinations, and applying complex transformations not supported natively by streaming sinks.
Question 7: In a Delta Lake streaming read (`spark.readStream.format('delta')`), what does setting `ignoreChanges=True` allow?
- Ignores schema changes in the Delta table
- Allows the stream to continue even when the source Delta table has data-changing operations like UPDATEs or DELETEs that would otherwise cause errors (Correct answer)
- Disables Change Data Feed on the source table
- Allows late-arriving data to bypass the watermark
Correct answer: Allows the stream to continue even when the source Delta table has data-changing operations like UPDATEs or DELETEs that would otherwise cause errors
By default, streaming reads from Delta fail if data-changing operations (UPDATEs, DELETEs, MERGE) affect the source; `ignoreChanges=True` allows the stream to continue by reprocessing affected files.
LFC Structured Streaming Pipelines 3
In Databricks, what is the difference between `trigger(processingTime='10 seconds')` and `trigger(once=True)` in Structured Streaming?