MS-DS Master of Data science Big Data Technologies 2

Question 1

Which Hadoop component is responsible for resource management and job scheduling across a cluster?

Accepted Answer

YARN

Answer

YARN (Yet Another Resource Negotiator) manages cluster resources and schedules jobs in Hadoop 2.x and later.

Question 2

In Apache Kafka, what is the role of a 'consumer group'?

Accepted Answer

A collection of consumers that jointly consume messages from a topic without duplication

Answer

A consumer group allows multiple consumers to divide topic partitions among themselves, ensuring each message is processed by only one member.

Question 3

What distinguishes Apache Flink's 'event time' processing from 'processing time' processing?

Accepted Answer

Event time uses the timestamp embedded in the data record; processing time uses the wall-clock time at the operator

Answer

Event time relies on timestamps within events, enabling correct results even for out-of-order or late-arriving data.

Question 4

Which storage format is columnar, supports predicate pushdown, and is commonly used in the Hadoop ecosystem for analytics?

Accepted Answer

Parquet

Answer

Parquet is a columnar storage format that enables efficient predicate pushdown and compression, making it ideal for analytical queries.

Question 5

In Apache Spark, what is the primary difference between a 'transformation' and an 'action'?

Accepted Answer

Transformations are lazy and return a new RDD; actions trigger computation and return a result

Answer

Transformations are lazy operations that define a new RDD lineage, while actions trigger actual computation and return values to the driver.

MS-DS Master of Data science Practice Test