CDP (Big Data Science) 2 — Questions and Answers
Question 1: Which CAP theorem property is sacrificed by Apache Cassandra by default in favor of availability and partition tolerance?
- Durability
- Consistency (Correct answer)
- Atomicity
- Isolation
Correct answer: Consistency
Cassandra is an AP system that prioritizes availability and partition tolerance, offering only eventual consistency by default.
Question 2: In a Lambda architecture, what is the primary role of the speed layer?
- Store historical batch data for long-term queries
- Provide low-latency views of recent data not yet processed by the batch layer (Correct answer)
- Merge batch and serving layer outputs into a unified view
- Compress and archive incoming streaming data
Correct answer: Provide low-latency views of recent data not yet processed by the batch layer
The speed layer processes recent data in real-time to compensate for the high latency of the batch layer.
Question 3: What does the 'embarrassingly parallel' term mean in the context of big data processing?
- Tasks require extensive inter-node communication
- Tasks can be divided into independent subtasks with no dependency between them (Correct answer)
- Only a single node can process the data at a time
- Tasks must be sequentially executed to maintain data order
Correct answer: Tasks can be divided into independent subtasks with no dependency between them
Embarrassingly parallel problems can be split into completely independent subtasks, requiring no communication between parallel workers.
Question 4: Which technique is used in Spark to avoid full data shuffles when joining a large dataset with a small one?
- Salting
- Broadcast join (Correct answer)
- Repartitioning
- Coalescing
Correct answer: Broadcast join
A broadcast join sends the smaller dataset to all worker nodes, eliminating the costly shuffle of the large dataset.
Question 5: In the context of streaming analytics, what is 'watermarking' used for?
- Encrypting data streams for security compliance
- Handling late-arriving data by defining a threshold for event-time delays (Correct answer)
- Marking data lineage across distributed pipelines
- Compressing stream checkpoints to reduce storage
Correct answer: Handling late-arriving data by defining a threshold for event-time delays
Watermarking sets a maximum tolerated delay for late data, allowing the system to finalize window computations despite out-of-order events.
Question 6: Which file format is optimized for columnar storage and is widely used in big data ecosystems for analytical queries?
- Avro
- JSON
- Parquet (Correct answer)
- CSV
Correct answer: Parquet
Parquet is a columnar storage format that enables efficient compression and encoding schemes, significantly improving analytical query performance.
Question 7: What is the primary purpose of data skew mitigation techniques like 'salting' in distributed big data joins?
- To encrypt sensitive join keys before transmission
- To distribute data more evenly across partitions and reduce hotspots (Correct answer)
- To cache frequently joined datasets in memory
- To convert wide joins into narrow transformations
Correct answer: To distribute data more evenly across partitions and reduce hotspots
Salting adds a random prefix to skewed keys to distribute them across multiple partitions, reducing the load on overloaded nodes.
CDP (Big Data Science) 2
Which CAP theorem property is sacrificed by Apache Cassandra by default in favor of availability and partition tolerance?