Data Engineering Distributed Data Processing 4

Question 1

What does 'speculative execution' do in a distributed batch framework?

Accepted Answer

Launches duplicate copies of slow tasks to finish faster

Answer

Speculative execution runs backup copies of straggler tasks and uses whichever finishes first.

Question 2

In Spark, persisting an RDD/DataFrame with caching mainly helps when:

Accepted Answer

It is reused across multiple actions

Answer

Caching avoids recomputation when the same dataset feeds several actions.

Question 3

What is the consequence of having too many small files on HDFS?

Accepted Answer

NameNode memory pressure from excessive metadata

Answer

Each file consumes NameNode metadata, so many tiny files strain the NameNode.

Question 4

Which technique mitigates skew by adding random prefixes to hot keys?

Accepted Answer

Salting

Answer

Salting spreads a hot key across multiple reducers by appending random values.

Question 5

In a stream processor, what is backpressure?

Accepted Answer

A slowdown signal when downstream cannot keep up

Answer

Backpressure throttles upstream producers when consumers fall behind, preventing overload.

Data Engineering Practice Test