MS-DS Master of Data science Big Data Technologies 2 — Questions and Answers
Question 1: Which Hadoop component is responsible for resource management and job scheduling across a cluster?
- HDFS
- YARN (Correct answer)
- MapReduce
- HBase
Correct answer: YARN
YARN (Yet Another Resource Negotiator) manages cluster resources and schedules jobs in Hadoop 2.x and later.
Question 2: In Apache Kafka, what is the role of a 'consumer group'?
- A set of brokers that replicate a topic partition
- A collection of consumers that jointly consume messages from a topic without duplication (Correct answer)
- A group of producers writing to the same topic
- A cluster of ZooKeeper nodes managing offsets
Correct answer: A collection of consumers that jointly consume messages from a topic without duplication
A consumer group allows multiple consumers to divide topic partitions among themselves, ensuring each message is processed by only one member.
Question 3: What distinguishes Apache Flink's 'event time' processing from 'processing time' processing?
- Event time uses the timestamp embedded in the data record; processing time uses the wall-clock time at the operator (Correct answer)
- Event time is faster because it skips ordering
- Processing time is more accurate for historical data
- Event time requires no watermarks
Correct answer: Event time uses the timestamp embedded in the data record; processing time uses the wall-clock time at the operator
Event time relies on timestamps within events, enabling correct results even for out-of-order or late-arriving data.
Question 4: Which storage format is columnar, supports predicate pushdown, and is commonly used in the Hadoop ecosystem for analytics?
- Avro
- JSON
- Parquet (Correct answer)
- CSV
Correct answer: Parquet
Parquet is a columnar storage format that enables efficient predicate pushdown and compression, making it ideal for analytical queries.
Question 5: In Apache Spark, what is the primary difference between a 'transformation' and an 'action'?
- Transformations write data to disk; actions keep data in memory
- Transformations are lazy and return a new RDD; actions trigger computation and return a result (Correct answer)
- Actions create new RDDs; transformations execute immediately
- There is no difference in Spark 3.x
Correct answer: Transformations are lazy and return a new RDD; actions trigger computation and return a result
Transformations are lazy operations that define a new RDD lineage, while actions trigger actual computation and return values to the driver.
Question 6: Which CAP theorem property does Apache Cassandra prioritize by default?
- Consistency and Partition tolerance (CP)
- Availability and Partition tolerance (AP) (Correct answer)
- Consistency and Availability (CA)
- All three equally
Correct answer: Availability and Partition tolerance (AP)
Cassandra is designed as an AP system, favoring availability and partition tolerance over strong consistency, though tunable consistency levels are available.
Question 7: What is the purpose of 'watermarks' in Apache Flink streaming applications?
- To encrypt data streams between nodes
- To signal the progress of event time and trigger window computations despite late data (Correct answer)
- To mark checkpoints for fault tolerance
- To set throughput limits on source connectors
Correct answer: To signal the progress of event time and trigger window computations despite late data
Watermarks indicate that no event with a timestamp earlier than the watermark value is expected, allowing Flink to close windows and emit results.
MS-DS Master of Data science Big Data Technologies 2
Which Hadoop component is responsible for resource management and job scheduling across a cluster?