MS-DS Master of Data science Big Data 4 — Questions and Answers
Question 1: Which Apache Flink concept tracks the progress of event time in a stream and triggers time-based operations?
- Checkpoints
- Watermarks (Correct answer)
- State backends
- Savepoints
Correct answer: Watermarks
Watermarks are monotonically increasing timestamps injected into the stream to signal that all events up to a certain event time have arrived.
Question 2: In the context of big data, what does 'data skew' refer to?
- Data encrypted asymmetrically
- Uneven distribution of data across partitions causing hotspots (Correct answer)
- Schema inconsistency across nodes
- Time-zone differences in timestamps
Correct answer: Uneven distribution of data across partitions causing hotspots
Data skew occurs when some partitions receive disproportionately more data, causing certain tasks to run much longer than others.
Question 3: Which storage abstraction in Apache Spark allows it to persist intermediate results and avoid recomputation?
- DAG scheduler
- RDD lineage
- Caching/persistence (Correct answer)
- Shuffle service
Correct answer: Caching/persistence
Spark's cache() and persist() methods store RDD or DataFrame data in memory (or disk) so reused datasets are not recomputed from scratch.
Question 4: A Kafka consumer group with 4 consumers subscribing to a topic with 3 partitions will result in:
- All 4 consumers reading all partitions
- 3 consumers active and 1 idle (Correct answer)
- 2 consumers sharing each partition
- An error due to mismatch
Correct answer: 3 consumers active and 1 idle
Kafka assigns at most one consumer per partition within a group, so one consumer will be idle when consumers outnumber partitions.
Question 5: Which technique resolves the problem of small files in HDFS degrading NameNode performance?
- Increasing replication factor
- Sequence files or HAR archives to consolidate small files (Correct answer)
- Adding more DataNodes
- Enabling HDFS federation
Correct answer: Sequence files or HAR archives to consolidate small files
Small files consume NameNode memory proportionally; combining them into Sequence Files or HAR archives reduces the number of metadata entries.
Question 6: What distinguishes a 'hot path' from a 'cold path' in big data architectures?
- Hot path uses SSDs; cold path uses HDDs
- Hot path processes data in near real-time; cold path processes historical data in batch (Correct answer)
- Hot path is encrypted; cold path is plaintext
- Hot path runs on-premises; cold path runs in the cloud
Correct answer: Hot path processes data in near real-time; cold path processes historical data in batch
The hot path handles streaming data for immediate insights while the cold path runs batch jobs over stored historical data.
Question 7: Which NoSQL data model is most appropriate for storing social network relationships with complex graph traversals?
- Document store
- Key-value store
- Column-family store
- Graph database (Correct answer)
Correct answer: Graph database
Graph databases like Neo4j natively represent entities as nodes and relationships as edges, enabling efficient multi-hop traversals.
MS-DS Master of Data science Big Data 4
Which Apache Flink concept tracks the progress of event time in a stream and triggers time-based operations?