Apache Spark Spark Streaming 2 — Questions and Answers
Question 1: What is checkpointing used for in Spark Streaming?
- To cache RDDs to disk for faster reuse
- To save the state and metadata of a streaming application for fault recovery (Correct answer)
- To snapshot DataFrames at regular intervals
- To synchronize streaming and batch jobs
Correct answer: To save the state and metadata of a streaming application for fault recovery
Checkpointing saves the streaming application's state (offsets, aggregation state) to fault-tolerant storage for recovery after failures.
Question 2: Which join type is supported in Spark Structured Streaming for stream-to-static dataset joins?
- Only inner joins
- Inner, left outer, right outer, and full outer joins (Correct answer)
- Only left outer joins
- No joins are supported in streaming
Correct answer: Inner, left outer, right outer, and full outer joins
Stream-static joins in Structured Streaming support inner, left outer, right outer, and full outer joins.
Question 3: What is the Complete output mode in Spark Structured Streaming?
- Writes only new rows since the last trigger
- Writes the entire updated result table to the sink after every trigger (Correct answer)
- Writes updated rows that changed since the last trigger
- Writes the final result when the stream ends
Correct answer: Writes the entire updated result table to the sink after every trigger
Complete mode writes the entire result table to the sink after every trigger, suitable for aggregations that update existing results.
Question 4: Which source reads data from Apache Kafka in Spark Structured Streaming?
- spark.readStream.format('kafka') (Correct answer)
- spark.readStream.format('kafkaSource')
- spark.kafka.read()
- spark.stream('kafka')
Correct answer: spark.readStream.format('kafka')
spark.readStream.format('kafka') reads streaming data from Kafka topics using the Kafka connector for Structured Streaming.
Question 5: What does the foreachBatch() sink do in Structured Streaming?
- Processes each micro-batch as a static DataFrame with a user-defined function (Correct answer)
- Iterates over each row in the stream individually
- Divides the stream into fixed-size batches for processing
- Writes each batch to a different output path
Correct answer: Processes each micro-batch as a static DataFrame with a user-defined function
foreachBatch() allows you to apply arbitrary operations on each micro-batch DataFrame, useful for writing to non-native sinks.
Question 6: In Spark Structured Streaming, what does event time refer to?
- The time when Spark processes a record
- The time when the record was generated at the source (Correct answer)
- The time when the record arrives at the Kafka topic
- The time when the sink writes the result
Correct answer: The time when the record was generated at the source
Event time is the time embedded in the data itself, representing when the event actually occurred at the source.
Apache Spark Spark Streaming 2
What is checkpointing used for in Spark Streaming?