Apache Spark Spark Streaming 2

Question 1

What is checkpointing used for in Spark Streaming?

Accepted Answer

To save the state and metadata of a streaming application for fault recovery

Answer

Checkpointing saves the streaming application's state (offsets, aggregation state) to fault-tolerant storage for recovery after failures.

Question 2

Which join type is supported in Spark Structured Streaming for stream-to-static dataset joins?

Accepted Answer

Inner, left outer, right outer, and full outer joins

Answer

Stream-static joins in Structured Streaming support inner, left outer, right outer, and full outer joins.

Question 3

What is the Complete output mode in Spark Structured Streaming?

Accepted Answer

Writes the entire updated result table to the sink after every trigger

Answer

Complete mode writes the entire result table to the sink after every trigger, suitable for aggregations that update existing results.

Question 4

Which source reads data from Apache Kafka in Spark Structured Streaming?

Accepted Answer

spark.readStream.format('kafka')

Answer

spark.readStream.format('kafka') reads streaming data from Kafka topics using the Kafka connector for Structured Streaming.

Question 5

What does the foreachBatch() sink do in Structured Streaming?

Accepted Answer

Processes each micro-batch as a static DataFrame with a user-defined function

Answer

foreachBatch() allows you to apply arbitrary operations on each micro-batch DataFrame, useful for writing to non-native sinks.

Apache Spark Practice Test