MS-DS Master of Data science Big Data Technologies 5

Question 1

In Apache Spark, what is the 'Catalyst optimizer' responsible for?

Accepted Answer

Transforming logical query plans into optimized physical execution plans for DataFrames and Datasets

Answer

Catalyst is Spark SQL's rule-based and cost-based query optimizer that rewrites logical plans into efficient physical plans.

Question 2

What distinguishes an 'append-only' log in Apache Kafka from a traditional message queue?

Accepted Answer

Kafka retains messages for a configurable retention period, allowing multiple consumers to replay events independently

Answer

Kafka's immutable log persists records beyond consumption, enabling multiple independent consumers and event replay without impacting each other.

Question 3

Which Hadoop ecosystem tool is best suited for running SQL-like queries directly on data stored in HDFS or S3 without moving it to a separate database?

Accepted Answer

Apache Hive

Answer

Apache Hive provides HiveQL, a SQL dialect that translates queries into MapReduce, Tez, or Spark jobs that run directly over data in HDFS or compatible object stores.

Question 4

In the context of big data pipelines, what is 'backpressure' and why does it matter?

Accepted Answer

A flow-control mechanism where a downstream component signals a slower ingestion rate to prevent being overwhelmed

Answer

Backpressure propagates downstream capacity constraints upstream, preventing faster producers from overwhelming slower consumers and causing out-of-memory failures.

Question 5

What is the purpose of Delta Lake's transaction log ('_delta_log' directory)?

Accepted Answer

It records every change to the table as a JSON commit entry, enabling ACID transactions and time-travel queries

Answer

Delta Lake's transaction log is an ordered list of atomic commits that provides ACID properties, optimistic concurrency control, and the ability to query previous table versions.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Big Data Technologies 5