MS-DS Master of Data science Big Data 5

Question 1

Which strategy reduces shuffle data volume in Spark by replacing groupByKey with a more efficient alternative?

Accepted Answer

Replacing groupByKey() with reduceByKey()

Answer

reduceByKey() applies a combining function locally on each partition before shuffling, dramatically reducing the data transferred across the network.

Question 2

In data lakehouse architecture, what role does a metadata layer like Apache Iceberg or Delta Lake serve?

Accepted Answer

It adds ACID transactions, schema evolution, and time travel to data lake files

Answer

Table formats like Iceberg and Delta Lake maintain transactional metadata over raw files, enabling ACID guarantees, schema evolution, and historical snapshots.

Question 3

Which metric best quantifies the throughput of a big data streaming system?

Accepted Answer

Events processed per second

Answer

Events (or messages) processed per second directly measures how much data the system handles over time, representing throughput.

Question 4

Apache Beam's unified programming model is designed to allow pipelines to run on multiple execution engines. What is the abstraction called that wraps these engines?

Accepted Answer

Runner

Answer

A Beam Runner (e.g., Dataflow Runner, Flink Runner) translates the portable Beam pipeline into the target engine's native execution model.

Question 5

In a distributed join, which optimization avoids a shuffle by sending a small table to all worker nodes?

Accepted Answer

Broadcast join

Answer

A broadcast join replicates the smaller table to every executor so the larger table can be joined locally without shuffling either dataset.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Big Data 5