(Big Data Science) Flashcards

Which CAP theorem property is sacrificed by Apache Cassandra by default in favor of availability and partition tolerance?

Answer: Consistency

Cassandra is an AP system that prioritizes availability and partition tolerance, offering only eventual consistency by default.

In a Lambda architecture, what is the primary role of the speed layer?

Answer: Provide low-latency views of recent data not yet processed by the batch layer

The speed layer processes recent data in real-time to compensate for the high latency of the batch layer.

What does the 'embarrassingly parallel' term mean in the context of big data processing?

Answer: Tasks can be divided into independent subtasks with no dependency between them

Embarrassingly parallel problems can be split into completely independent subtasks, requiring no communication between parallel workers.

Which technique is used in Spark to avoid full data shuffles when joining a large dataset with a small one?

Answer: Broadcast join

A broadcast join sends the smaller dataset to all worker nodes, eliminating the costly shuffle of the large dataset.

In the context of streaming analytics, what is 'watermarking' used for?

Answer: Handling late-arriving data by defining a threshold for event-time delays

Watermarking sets a maximum tolerated delay for late data, allowing the system to finalize window computations despite out-of-order events.

Which file format is optimized for columnar storage and is widely used in big data ecosystems for analytical queries?

Answer: Parquet

Parquet is a columnar storage format that enables efficient compression and encoding schemes, significantly improving analytical query performance.

What is the primary purpose of data skew mitigation techniques like 'salting' in distributed big data joins?

Answer: To distribute data more evenly across partitions and reduce hotspots

Salting adds a random prefix to skewed keys to distribute them across multiple partitions, reducing the load on overloaded nodes.