Apache Spark Spark Architecture and Cluster Management 2

Question 1

What does deploy mode 'cluster' mean in Spark?

Accepted Answer

The Driver process runs on one of the worker nodes managed by the cluster manager

Answer

In cluster deploy mode, the Driver runs on one of the worker nodes in the cluster, which is preferred for production jobs.

Question 2

What is dynamic resource allocation in Spark?

Accepted Answer

Scaling the number of executors up and down based on workload

Answer

Dynamic resource allocation allows Spark to add executors when tasks are queued and remove idle executors to free cluster resources.

Question 3

What is the purpose of the external shuffle service in Spark?

Accepted Answer

Maintains shuffle files on worker nodes so executors can be released during dynamic allocation

Answer

The external shuffle service runs on worker nodes and holds shuffle files, allowing executors to be released while preserving shuffle data.

Question 4

What does spark-submit --master yarn --deploy-mode cluster do?

Accepted Answer

Submits a Spark application to a YARN cluster with the driver running on a YARN container

Answer

This command submits a Spark application to YARN and runs the Driver in a YARN ApplicationMaster container on the cluster.

Question 5

What is YARN in the context of Apache Spark?

Accepted Answer

A cluster resource manager that allocates containers for Spark executors and the driver

Answer

YARN (Yet Another Resource Negotiator) is a Hadoop resource manager that Spark uses to allocate containers for executors and the application master.

Apache Spark Practice Test