LFC Apache Spark Data Engineering 3 — Questions and Answers
Question 1: Which Spark API abstraction represents an immutable, distributed collection of objects with no schema enforcement?
- DataFrame
- Dataset
- RDD (Correct answer)
- DynamicFrame
Correct answer: RDD
RDD (Resilient Distributed Dataset) is the lowest-level Spark abstraction — distributed and fault-tolerant but without schema or query optimization.
Question 2: In Delta Lake's transaction log, what file type records each committed transaction's metadata and statistics?
- .parquet checkpoint file
- JSON entry in the _delta_log directory (Correct answer)
- Manifest file in the table root
- Avro schema file
Correct answer: JSON entry in the _delta_log directory
Every Delta Lake commit writes a JSON file to the _delta_log directory recording the operations, file adds/removes, and column statistics.
Question 3: What Spark technique reduces data movement by pushing filter conditions down to the data source before loading into memory?
- Broadcast join
- Predicate pushdown (Correct answer)
- Adaptive query execution
- Whole-stage code generation
Correct answer: Predicate pushdown
Predicate pushdown allows Spark to pass filter conditions to the storage layer (e.g., Parquet or Delta) so only matching rows are read.
Question 4: When should you use `df.cache()` versus `df.persist(StorageLevel.DISK_ONLY)` in Spark?
- cache() for DataFrames, persist() for RDDs only
- cache() stores in memory only; persist() allows specifying storage level including disk (Correct answer)
- They are functionally identical
- persist() is deprecated in Spark 3.x
Correct answer: cache() stores in memory only; persist() allows specifying storage level including disk
cache() is shorthand for persist(MEMORY_AND_DISK) in DataFrames, while persist() lets you specify storage levels like DISK_ONLY, MEMORY_ONLY, or OFF_HEAP.
Question 5: In a Databricks lakehouse, what is the role of the Unity Catalog?
- A UI for visualizing Spark DAGs
- A centralized governance layer for data, AI assets, and fine-grained access control across workspaces (Correct answer)
- A query engine for ad-hoc SQL
- A file compression utility for Delta tables
Correct answer: A centralized governance layer for data, AI assets, and fine-grained access control across workspaces
Unity Catalog provides centralized governance, lineage tracking, and fine-grained access control for tables, volumes, and ML models across multiple Databricks workspaces.
Question 6: Which Spark join strategy avoids a shuffle by broadcasting the smaller table to all executor nodes?
- Sort-merge join
- Shuffle hash join
- Broadcast hash join (Correct answer)
- Nested loop join
Correct answer: Broadcast hash join
Broadcast hash join sends a copy of the smaller table to every executor, allowing the join to be performed locally without shuffling the larger table.
Question 7: What does Spark's Adaptive Query Execution (AQE) do at runtime?
- Rewrites SQL queries before parsing
- Dynamically adjusts the query plan based on runtime statistics collected during execution (Correct answer)
- Allocates GPU resources for ML workloads
- Compresses intermediate shuffle data automatically
Correct answer: Dynamically adjusts the query plan based on runtime statistics collected during execution
AQE re-optimizes query plans at runtime using actual shuffle partition sizes and join input statistics, enabling coalescing partitions and switching join strategies.
LFC Apache Spark Data Engineering 3
Which Spark API abstraction represents an immutable, distributed collection of objects with no schema enforcement?