Apache Spark Spark Core and RDDs 2 — Questions and Answers
Question 1: Which method is used to persist an RDD in Spark?
- cache() or persist() (Correct answer)
- store()
- save()
- memorize()
Correct answer: cache() or persist()
cache() and persist() are used to persist an RDD; cache() is a shorthand for persist(StorageLevel.MEMORY_ONLY).
Question 2: What is a narrow transformation in Spark?
- A transformation that requires data shuffling
- A transformation where each input partition contributes to only one output partition (Correct answer)
- A transformation applied to a narrow dataset
- A transformation that reduces the number of partitions
Correct answer: A transformation where each input partition contributes to only one output partition
Narrow transformations are those where each input partition maps to only one output partition, requiring no data shuffle.
Question 3: Which of the following is a wide transformation (shuffle) in Spark?
- filter()
- map()
- groupByKey() (Correct answer)
- union()
Correct answer: groupByKey()
groupByKey() is a wide transformation because it requires shuffling data across partitions to group all values for each key.
Question 4: What is lineage in Apache Spark?
- The order in which jobs are executed
- The record of transformations used to build an RDD from base data (Correct answer)
- The dependency tree of Spark executors
- The log of actions performed on a cluster
Correct answer: The record of transformations used to build an RDD from base data
Lineage is the sequence of transformations that were applied to create an RDD, allowing Spark to recompute lost partitions.
Question 5: Which Spark RDD action returns the first n elements of the RDD?
- first(n)
- head(n)
- take(n) (Correct answer)
- top(n)
Correct answer: take(n)
take(n) returns the first n elements of the RDD to the driver program.
Question 6: How does Spark achieve fault tolerance with RDDs?
- By replicating data across multiple nodes
- By recomputing lost partitions using lineage information (Correct answer)
- By writing all intermediate data to disk
- By maintaining checkpoints after every transformation
Correct answer: By recomputing lost partitions using lineage information
Spark achieves fault tolerance by recomputing lost RDD partitions using the recorded lineage of transformations.
Apache Spark Spark Core and RDDs 2
Which method is used to persist an RDD in Spark?