MS-DS Master of Data science Master of Data science Big Data Technologies 1 — Questions and Answers
Question 1: Which Apache framework provides a unified analytics engine that supports both batch processing and interactive SQL queries through its Catalyst optimizer?
- Apache Flink
- Apache Hive
- Apache Spark SQL (Correct answer)
- Apache Storm
Correct answer: Apache Spark SQL
Apache Spark SQL uses the Catalyst optimizer to translate SQL queries and DataFrame operations into optimized execution plans, supporting both batch and interactive workloads within the Spark ecosystem.
Question 2: In the context of Apache Kafka, what is the term for an ordered, immutable sequence of records that is continually appended to?
- Queue
- Topic
- Partition
- Log (Correct answer)
Correct answer: Log
Kafka's core abstraction is the log — an ordered, immutable sequence of records. Topics are categories, and partitions are ordered subdivisions of a topic, but the fundamental data structure is the log.
Question 3: Which consistency model does Apache Cassandra use by default to balance availability and partition tolerance in distributed deployments?
- Strong consistency
- Eventual consistency (Correct answer)
- Linearizability
- Causal consistency
Correct answer: Eventual consistency
Cassandra is designed around the CAP theorem favoring availability and partition tolerance (AP), using eventual consistency by default, meaning all replicas will converge to the same value given enough time without new updates.
Question 4: What technique does Apache HBase use to physically organize data on disk, enabling fast random read and write access to large datasets?
- Row-oriented flat files
- LSM trees (Log-Structured Merge-trees) (Correct answer)
- B+ tree indexes
- Hash partitioning
Correct answer: LSM trees (Log-Structured Merge-trees)
HBase uses LSM trees, which buffer writes in memory (MemStore) and periodically flush them to sorted on-disk files (HFiles), enabling high write throughput while supporting efficient range scans.
Question 5: In Apache Spark's structured streaming, what abstraction treats a live data stream as an unbounded table that is continuously updated?
- DStream
- RDD
- DataFrame (Correct answer)
- Continuous DataFrame
Correct answer: DataFrame
Spark Structured Streaming models streaming data as an unbounded DataFrame/Dataset, allowing developers to write batch-like DataFrame queries that execute incrementally as new data arrives, replacing the older DStream API.
Question 6: Which data serialization format, commonly used in big data pipelines, provides schema evolution support and compact binary encoding without requiring code generation?
- Protocol Buffers
- Apache Avro (Correct answer)
- Apache Thrift
- MessagePack
Correct answer: Apache Avro
Apache Avro stores the schema alongside the data in JSON format and serializes the payload in compact binary, enabling schema evolution (adding/removing fields with defaults) without code generation, making it popular in Kafka and Hadoop ecosystems.
MS-DS Master of Data science Master of Data science Big Data Technologies 1
Which Apache framework provides a unified analytics engine that supports both batch processing and interactive SQL queries through its Catalyst optimizer?