Apache Spark Spark Streaming 1 — Questions and Answers
Question 1: What is the fundamental abstraction in Spark Structured Streaming?
- DStream
- RDD
- Unbounded DataFrame/Dataset (Correct answer)
- StreamRDD
Correct answer: Unbounded DataFrame/Dataset
Structured Streaming treats a live data stream as an unbounded table (DataFrame/Dataset) that is continuously appended.
Question 2: What is a DStream in Spark Streaming?
- A distributed stream of DataFrames
- A sequence of RDDs representing a continuous stream of data (Correct answer)
- A real-time SQL query engine
- A fault-tolerant message queue
Correct answer: A sequence of RDDs representing a continuous stream of data
A DStream (Discretized Stream) is a continuous sequence of RDDs, each representing data from a time interval.
Question 3: Which output mode in Structured Streaming writes only new rows added to the result table since the last trigger?
- Complete
- Update
- Append (Correct answer)
- Incremental
Correct answer: Append
Append mode writes only new rows appended to the result table since the last trigger to the sink.
Question 4: What is a watermark in Spark Structured Streaming?
- A threshold defining the maximum allowed lateness for event-time data (Correct answer)
- A checkpoint marker for fault recovery
- A memory limit for streaming aggregations
- A trigger interval for micro-batch processing
Correct answer: A threshold defining the maximum allowed lateness for event-time data
A watermark specifies how late event-time data can be and still be included in the computation, allowing Spark to drop old state.
Question 5: Which of the following is a valid trigger for Structured Streaming in Spark?
- ProcessingTime trigger (Correct answer)
- EventTime trigger
- BatchSize trigger
- RowCount trigger
Correct answer: ProcessingTime trigger
ProcessingTime trigger runs micro-batches at a specified interval, e.g., Trigger.ProcessingTime('10 seconds').
Question 6: In Spark Structured Streaming, which sink writes results to an in-memory table for interactive querying?
- console
- file
- memory (Correct answer)
- foreach
Correct answer: memory
The memory sink stores the output in-memory as a named table, suitable for testing and interactive queries.
Apache Spark Spark Streaming 1
What is the fundamental abstraction in Spark Structured Streaming?