Apache Spark Spark Core and RDDs 3

Question 1

What is the purpose of the reduceByKey() transformation in Spark?

Accepted Answer

Aggregates values for each key using a given function

Answer

reduceByKey() merges values for each key using an associative and commutative function, performing a local combine before shuffling.

Question 2

Which of these creates an RDD from an existing Scala/Python collection in Spark?

Accepted Answer

sc.parallelize()

Answer

sc.parallelize() creates an RDD by distributing an existing local collection across the cluster.

Question 3

What is the role of the SparkContext in a Spark application?

Accepted Answer

Serves as the entry point for Spark functionality and represents the connection to a Spark cluster

Answer

SparkContext is the main entry point for Spark Core API, representing the connection to a Spark cluster and used to create RDDs.

Question 4

What does the coalesce() transformation do in Spark?

Accepted Answer

Reduces the number of partitions with minimal data shuffling

Answer

coalesce() reduces the number of partitions by merging them, avoiding a full shuffle when decreasing partition count.

Question 5

Which action computes the sum of all elements in a numeric RDD?

Accepted Answer

reduce(lambda a,b: a+b)

Answer

reduce() with a lambda that adds two values computes the sum across all elements in the RDD.

Apache Spark Practice Test