Apache Spark Spark Architecture and Cluster Management 2 — Questions and Answers
Question 1: What does deploy mode 'cluster' mean in Spark?
- The Spark application runs on every node in the cluster
- The Driver process runs on one of the worker nodes managed by the cluster manager (Correct answer)
- The application is replicated across multiple driver nodes for high availability
- The cluster manager selects the optimal executor placement
Correct answer: The Driver process runs on one of the worker nodes managed by the cluster manager
In cluster deploy mode, the Driver runs on one of the worker nodes in the cluster, which is preferred for production jobs.
Question 2: What is dynamic resource allocation in Spark?
- Automatically adjusting executor memory based on task requirements
- Scaling the number of executors up and down based on workload (Correct answer)
- Distributing data across nodes based on access frequency
- Automatically partitioning data based on cluster size
Correct answer: Scaling the number of executors up and down based on workload
Dynamic resource allocation allows Spark to add executors when tasks are queued and remove idle executors to free cluster resources.
Question 3: What is the purpose of the external shuffle service in Spark?
- Handles shuffle operations faster than in-executor processing
- Maintains shuffle files on worker nodes so executors can be released during dynamic allocation (Correct answer)
- Streams shuffle data directly between executors bypassing the driver
- Compresses shuffle data to reduce network bandwidth usage
Correct answer: Maintains shuffle files on worker nodes so executors can be released during dynamic allocation
The external shuffle service runs on worker nodes and holds shuffle files, allowing executors to be released while preserving shuffle data.
Question 4: What does spark-submit --master yarn --deploy-mode cluster do?
- Submits a Spark application to a YARN cluster with the driver running on a YARN container (Correct answer)
- Submits a Spark application to a standalone Spark cluster in cluster mode
- Runs a Spark job on the local machine using all available cores
- Deploys a Spark application to every node in the YARN cluster simultaneously
Correct answer: Submits a Spark application to a YARN cluster with the driver running on a YARN container
This command submits a Spark application to YARN and runs the Driver in a YARN ApplicationMaster container on the cluster.
Question 5: What is YARN in the context of Apache Spark?
- A data serialization format used in Spark
- A cluster resource manager that allocates containers for Spark executors and the driver (Correct answer)
- A distributed file system for storing Spark data
- A scheduler plugin for Spark's DAGScheduler
Correct answer: A cluster resource manager that allocates containers for Spark executors and the driver
YARN (Yet Another Resource Negotiator) is a Hadoop resource manager that Spark uses to allocate containers for executors and the application master.
Question 6: How does Spark handle executor failures during a job?
- The job fails immediately when any executor is lost
- Spark re-schedules the failed tasks on other available executors using RDD lineage (Correct answer)
- The driver takes over the computations from the failed executor
- Spark pauses the job and waits for the executor to recover
Correct answer: Spark re-schedules the failed tasks on other available executors using RDD lineage
When an executor fails, Spark reschedules its tasks on other executors and recomputes any lost RDD partitions using lineage.
Apache Spark Spark Architecture and Cluster Management 2
What does deploy mode 'cluster' mean in Spark?