Apache Spark Spark Architecture and Cluster Management

Question 1

What is the role of the Spark Driver in a Spark application?

Accepted Answer

Hosts the SparkContext, maintains the DAG, and coordinates task scheduling across executors

Answer

The Driver program runs the main() function, creates SparkContext, builds the DAG, and communicates with the cluster manager to schedule tasks.

Question 2

What are Spark Executors?

Accepted Answer

JVM processes launched by the cluster manager that execute tasks and store RDD data

Answer

Executors are JVM processes running on worker nodes that execute tasks assigned by the Driver and cache RDD partitions in memory.

Question 3

Which cluster manager is natively built into Apache Spark for standalone deployments?

Accepted Answer

Spark Standalone Mode

Answer

Spark's built-in standalone cluster manager provides a simple way to deploy Spark without YARN, Mesos, or Kubernetes.

Question 4

What is a Spark Stage?

Accepted Answer

A phase of the DAG separated by wide transformations (shuffle boundaries)

Answer

A Stage is a set of tasks that can be computed without shuffling data; stage boundaries are determined by wide transformations.

Question 5

What is a Spark Task?

Accepted Answer

A unit of work sent to one executor to process one data partition

Answer

A Task is the smallest unit of work in Spark, processing one data partition and running on a single executor core.

Apache Spark Practice Test