Apache Spark Spark Architecture and Cluster Management 1 — Questions and Answers
Question 1: What is the role of the Spark Driver in a Spark application?
- Executes the actual task computations on data partitions
- Hosts the SparkContext, maintains the DAG, and coordinates task scheduling across executors (Correct answer)
- Manages cluster resources and allocates containers
- Stores intermediate shuffle data between stages
Correct answer: Hosts the SparkContext, maintains the DAG, and coordinates task scheduling across executors
The Driver program runs the main() function, creates SparkContext, builds the DAG, and communicates with the cluster manager to schedule tasks.
Question 2: What are Spark Executors?
- JVM processes launched by the cluster manager that execute tasks and store RDD data (Correct answer)
- Worker nodes in the cluster that manage data replication
- Processes on the driver node that compile the DAG into tasks
- Memory managers that handle executor JVM garbage collection
Correct answer: JVM processes launched by the cluster manager that execute tasks and store RDD data
Executors are JVM processes running on worker nodes that execute tasks assigned by the Driver and cache RDD partitions in memory.
Question 3: Which cluster manager is natively built into Apache Spark for standalone deployments?
- YARN
- Mesos
- Kubernetes
- Spark Standalone Mode (Correct answer)
Correct answer: Spark Standalone Mode
Spark's built-in standalone cluster manager provides a simple way to deploy Spark without YARN, Mesos, or Kubernetes.
Question 4: What is a Spark Stage?
- A group of tasks that can be executed in parallel without a shuffle
- A single task running on one partition
- A phase of the DAG separated by wide transformations (shuffle boundaries) (Correct answer)
- A container for multiple Spark jobs
Correct answer: A phase of the DAG separated by wide transformations (shuffle boundaries)
A Stage is a set of tasks that can be computed without shuffling data; stage boundaries are determined by wide transformations.
Question 5: What is a Spark Task?
- A complete Spark job submitted by the driver
- A unit of work sent to one executor to process one data partition (Correct answer)
- A scheduled operation that runs on a timer
- A sub-DAG of transformations within a single stage
Correct answer: A unit of work sent to one executor to process one data partition
A Task is the smallest unit of work in Spark, processing one data partition and running on a single executor core.
Question 6: In Spark's DAG (Directed Acyclic Graph), what does each node represent?
- An executor JVM process
- An RDD or DataFrame partition
- An RDD or DataFrame with transformations applied (Correct answer)
- A shuffle exchange operation
Correct answer: An RDD or DataFrame with transformations applied
In Spark's DAG, each node represents an RDD or DataFrame, and edges represent transformations applied between them.
Apache Spark Spark Architecture and Cluster Management 1
What is the role of the Spark Driver in a Spark application?