MS-DS Master of Data science Big Data 4

Question 1

Which Apache Flink concept tracks the progress of event time in a stream and triggers time-based operations?

Accepted Answer

Watermarks

Answer

Watermarks are monotonically increasing timestamps injected into the stream to signal that all events up to a certain event time have arrived.

Question 2

In the context of big data, what does 'data skew' refer to?

Accepted Answer

Uneven distribution of data across partitions causing hotspots

Answer

Data skew occurs when some partitions receive disproportionately more data, causing certain tasks to run much longer than others.

Question 3

Which storage abstraction in Apache Spark allows it to persist intermediate results and avoid recomputation?

Accepted Answer

Caching/persistence

Answer

Spark's cache() and persist() methods store RDD or DataFrame data in memory (or disk) so reused datasets are not recomputed from scratch.

Question 4

A Kafka consumer group with 4 consumers subscribing to a topic with 3 partitions will result in:

Accepted Answer

3 consumers active and 1 idle

Answer

Kafka assigns at most one consumer per partition within a group, so one consumer will be idle when consumers outnumber partitions.

Question 5

Which technique resolves the problem of small files in HDFS degrading NameNode performance?

Accepted Answer

Sequence files or HAR archives to consolidate small files

Answer

Small files consume NameNode memory proportionally; combining them into Sequence Files or HAR archives reduces the number of metadata entries.

MS-DS Master of Data science Practice Test