CDP (Big Data Science) 3 — Questions and Answers
Question 1: Which consensus algorithm is commonly used by distributed systems like ZooKeeper to achieve leader election and coordination?
- Raft
- Paxos (Correct answer)
- Two-Phase Commit
- Gossip Protocol
Correct answer: Paxos
ZooKeeper uses the ZAB (ZooKeeper Atomic Broadcast) protocol, which is closely based on the Paxos consensus algorithm.
Question 2: In Hadoop MapReduce, what happens during the 'shuffle and sort' phase?
- Input data is split into fixed-size blocks and distributed to mappers
- Mapper output is transferred to reducers and sorted by key before reduction (Correct answer)
- The final output is written to HDFS in sorted order
- Combiners aggregate all intermediate results before shuffle begins
Correct answer: Mapper output is transferred to reducers and sorted by key before reduction
During shuffle and sort, the framework transfers mapper outputs to the appropriate reducers and sorts them by key so each reducer receives all values for a given key.
Question 3: What is a 'hot partition' problem in distributed NoSQL databases?
- A partition that is stored on nodes with high CPU temperature causing data loss
- A single partition receiving a disproportionately high volume of read/write requests (Correct answer)
- A partition that has exceeded its maximum TTL and must be expired
- A partition that stores time-series data sorted in descending order
Correct answer: A single partition receiving a disproportionately high volume of read/write requests
A hot partition occurs when one partition handles significantly more traffic than others, creating a performance bottleneck in the distributed system.
Question 4: Which characteristic of big data refers to the trustworthiness and accuracy of the data collected?
- Velocity
- Volume
- Veracity (Correct answer)
- Variety
Correct answer: Veracity
Veracity refers to the uncertainty, noise, and reliability of data — ensuring data quality and accuracy in big data systems.
Question 5: In Apache Kafka, what is the role of the 'consumer group'?
- To replicate topic partitions across multiple brokers for durability
- To allow multiple consumers to share the work of reading a topic's partitions in parallel (Correct answer)
- To batch-produce messages to a topic from multiple producer threads
- To compress messages across partitions for storage efficiency
Correct answer: To allow multiple consumers to share the work of reading a topic's partitions in parallel
A consumer group allows multiple consumers to each read from a distinct subset of partitions, enabling parallel consumption of a topic.
Question 6: What is the purpose of using 'approximate algorithms' like HyperLogLog in big data analytics?
- To provide exact counts with lower memory overhead than hash maps
- To estimate metrics like cardinality with significantly reduced memory and computation cost (Correct answer)
- To parallelize exact aggregations across distributed nodes
- To replace machine learning models with rule-based approximations
Correct answer: To estimate metrics like cardinality with significantly reduced memory and computation cost
HyperLogLog and similar probabilistic algorithms trade a small accuracy loss for dramatic reductions in memory usage when estimating cardinality at scale.
Question 7: Which property of HDFS (Hadoop Distributed File System) makes it fault-tolerant?
- It uses RAID storage on each DataNode for hardware redundancy
- It stores multiple replicas of each data block across different DataNodes (Correct answer)
- It uses synchronous writes to a backup NameNode before acknowledging writes
- It encrypts all blocks and stores checksums on the NameNode
Correct answer: It stores multiple replicas of each data block across different DataNodes
HDFS achieves fault tolerance by replicating each block (typically 3 times) across multiple DataNodes, ensuring data availability if nodes fail.
CDP (Big Data Science) 3
Which consensus algorithm is commonly used by distributed systems like ZooKeeper to achieve leader election and coordination?