MS-DS Master of Data science Master of Data science Big Data Technologies 1

Question 1

Which Apache framework provides a unified analytics engine that supports both batch processing and interactive SQL queries through its Catalyst optimizer?

Accepted Answer

Apache Spark SQL

Answer

Apache Spark SQL uses the Catalyst optimizer to translate SQL queries and DataFrame operations into optimized execution plans, supporting both batch and interactive workloads within the Spark ecosystem.

Question 2

In the context of Apache Kafka, what is the term for an ordered, immutable sequence of records that is continually appended to?

Accepted Answer

Log

Answer

Kafka's core abstraction is the log — an ordered, immutable sequence of records. Topics are categories, and partitions are ordered subdivisions of a topic, but the fundamental data structure is the log.

Question 3

Which consistency model does Apache Cassandra use by default to balance availability and partition tolerance in distributed deployments?

Accepted Answer

Eventual consistency

Answer

Cassandra is designed around the CAP theorem favoring availability and partition tolerance (AP), using eventual consistency by default, meaning all replicas will converge to the same value given enough time without new updates.

Question 4

What technique does Apache HBase use to physically organize data on disk, enabling fast random read and write access to large datasets?

Accepted Answer

LSM trees (Log-Structured Merge-trees)

Answer

HBase uses LSM trees, which buffer writes in memory (MemStore) and periodically flush them to sorted on-disk files (HFiles), enabling high write throughput while supporting efficient range scans.

Question 5

In Apache Spark's structured streaming, what abstraction treats a live data stream as an unbounded table that is continuously updated?

Accepted Answer

DataFrame

Answer

Spark Structured Streaming models streaming data as an unbounded DataFrame/Dataset, allowing developers to write batch-like DataFrame queries that execute incrementally as new data arrives, replacing the older DStream API.

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Practice Test

MS-DS Master of Data science Master of Data science Big Data Technologies 1