Master of Data science Big Data Technologies 1 Flashcards
6 cards from real MS-DS Master of Data science practice questions. Tap to flip, then mark Knew It or Still Learning — missed cards come back until you master them.
Read the first 6 Master of Data science Big Data Technologies 1 flashcards as text
Which Apache framework provides a unified analytics engine that supports both batch processing and interactive SQL queries through its Catalyst optimizer?
Answer: Apache Spark SQL
Apache Spark SQL uses the Catalyst optimizer to translate SQL queries and DataFrame operations into optimized execution plans, supporting both batch and interactive workloads within the Spark ecosystem.
In the context of Apache Kafka, what is the term for an ordered, immutable sequence of records that is continually appended to?
Answer: Log
Kafka's core abstraction is the log — an ordered, immutable sequence of records. Topics are categories, and partitions are ordered subdivisions of a topic, but the fundamental data structure is the log.
Which consistency model does Apache Cassandra use by default to balance availability and partition tolerance in distributed deployments?
Answer: Eventual consistency
Cassandra is designed around the CAP theorem favoring availability and partition tolerance (AP), using eventual consistency by default, meaning all replicas will converge to the same value given enough time without new updates.
What technique does Apache HBase use to physically organize data on disk, enabling fast random read and write access to large datasets?
Answer: LSM trees (Log-Structured Merge-trees)
HBase uses LSM trees, which buffer writes in memory (MemStore) and periodically flush them to sorted on-disk files (HFiles), enabling high write throughput while supporting efficient range scans.
In Apache Spark's structured streaming, what abstraction treats a live data stream as an unbounded table that is continuously updated?
Answer: DataFrame
Spark Structured Streaming models streaming data as an unbounded DataFrame/Dataset, allowing developers to write batch-like DataFrame queries that execute incrementally as new data arrives, replacing the older DStream API.
Which data serialization format, commonly used in big data pipelines, provides schema evolution support and compact binary encoding without requiring code generation?
Answer: Apache Avro
Apache Avro stores the schema alongside the data in JSON format and serializes the payload in compact binary, enabling schema evolution (adding/removing fields with defaults) without code generation, making it popular in Kafka and Hadoop ecosystems.