Data Engineering Distributed Data Processing 2

Question 1

In a MapReduce job, what is the primary purpose of the shuffle phase?

Accepted Answer

To group and transfer mapper output to reducers by key

Answer

To split the input file into blocks

Answer

To compress the final output

Answer

To launch the driver program

Question 2

What problem does data skew most directly cause in a distributed join?

Accepted Answer

A few overloaded tasks slow the whole stage

Answer

Loss of data replicas

Answer

Network encryption failures

Answer

Schema corruption

Question 3

Which Spark operation triggers actual computation rather than just building the DAG?

Accepted Answer

count()

Answer

filter()

Answer

map()

Answer

select()

Question 4

A broadcast join is most appropriate when:

Accepted Answer

One table is small enough to fit in each executor's memory

Answer

Both tables are extremely large

Answer

The join key is null for most rows

Answer

No join key exists

Question 5

What does HDFS block replication primarily provide?

Accepted Answer

Fault tolerance against node failure

Answer

Faster CPU scheduling

Answer

Lower storage cost

Answer

Stronger SQL typing

Question 6

In Spark, what is a 'narrow' transformation?

Accepted Answer

Each output partition depends on one input partition

Answer

Output depends on all partitions

Answer

It always triggers a shuffle

Answer

It writes directly to disk

Question 7

Why is checkpointing useful in long-running streaming jobs?

Accepted Answer

It persists state so the job can recover after failure

Answer

It compresses messages

Answer

It increases parallelism automatically

Answer

It removes the need for offsets