MS-DS Master of Data science Big Data Technologies 4

Question 1

Which Apache Spark optimization technique avoids shuffling by co-partitioning two datasets on the same key?

Accepted Answer

Sort-merge join with bucketing

Answer

Bucketing pre-partitions data on a join key so that matching partitions are co-located, eliminating the shuffle step during joins.

Question 2

In Google's original MapReduce paper, what is the purpose of the 'combiner' function?

Accepted Answer

To perform partial aggregation on the mapper's output before it is sent to the reducer, reducing network traffic

Answer

The combiner acts as a mini-reducer on each mapper node, aggregating intermediate key-value pairs locally to minimize data shuffled over the network.

Question 3

What is Apache Iceberg's primary advantage over a traditional Hive partitioned table?

Accepted Answer

Iceberg provides ACID transactions, hidden partitioning, and time-travel on large analytic tables without full table scans on metadata

Answer

Iceberg adds a metadata layer that enables ACID guarantees, partition evolution, and time-travel queries without modifying the underlying Parquet/ORC files.

Question 4

Which consistency model does Apache Cassandra use when the consistency level is set to 'QUORUM'?

Accepted Answer

A majority (more than half) of replicas in the replication factor must acknowledge

Answer

QUORUM requires acknowledgment from a majority of replicas, balancing consistency and availability.

Question 5

In stream processing, what does 'exactly-once semantics' guarantee?

Accepted Answer

Each record is processed exactly once, with no duplicates and no data loss

Answer

Exactly-once semantics ensures every record affects the output exactly one time, combining idempotent writes with transactional commits or distributed snapshots.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Big Data Technologies 4