Data Engineering Distributed Data Processing 2

Question 1

In a MapReduce job, what is the primary purpose of the shuffle phase?

Accepted Answer

To group and transfer mapper output to reducers by key

Answer

The shuffle phase sorts mapper output and routes records with the same key to the same reducer.

Question 2

What problem does data skew most directly cause in a distributed join?

Accepted Answer

A few overloaded tasks slow the whole stage

Answer

Skew concentrates many records on one key, making a few tasks far slower than the rest.

Question 3

Which Spark operation triggers actual computation rather than just building the DAG?

Accepted Answer

count()

Answer

count() is an action, while filter, map, and select are lazy transformations.

Question 4

A broadcast join is most appropriate when:

Accepted Answer

One table is small enough to fit in each executor's memory

Answer

Broadcasting a small table to every node avoids shuffling the large table.

Question 5

What does HDFS block replication primarily provide?

Accepted Answer

Fault tolerance against node failure

Answer

Multiple replicas let the cluster recover data if a DataNode fails.

Data Engineering Practice Test