LFC Apache Spark Data Engineering 2 — Questions and Answers
Question 1: Which Spark component is responsible for translating DataFrame operations into an optimized physical execution plan?
- Spark Scheduler
- Catalyst Optimizer (Correct answer)
- Tungsten Engine
- DAG Visualizer
Correct answer: Catalyst Optimizer
The Catalyst Optimizer analyzes logical plans and applies rule-based and cost-based optimizations to generate an efficient physical plan.
Question 2: In Delta Lake, what mechanism prevents two concurrent write operations from corrupting the same table?
- Row-level locking
- Optimistic concurrency control (Correct answer)
- Pessimistic locking
- Two-phase commit protocol
Correct answer: Optimistic concurrency control
Delta Lake uses optimistic concurrency control, detecting conflicts at commit time and retrying or failing conflicting transactions.
Question 3: What does the `spark.sql.shuffle.partitions` configuration control?
- Number of input file splits
- Number of partitions after a wide transformation like join or aggregation (Correct answer)
- Memory allocated per executor
- Number of cores per task
Correct answer: Number of partitions after a wide transformation like join or aggregation
This setting determines how many partitions Spark creates during shuffle operations such as groupBy, join, and distinct.
Question 4: Which file format used in lakehouses stores data in columnar format with built-in schema and statistics per row group?
- CSV
- Avro
- Parquet (Correct answer)
- ORC
Correct answer: Parquet
Parquet stores data columnar with per-column statistics per row group, enabling predicate pushdown and efficient compression.
Question 5: When using Spark Structured Streaming, what does the `trigger(availableNow=True)` option do?
- Runs the stream continuously with micro-batches
- Processes all available data then stops, like a batch job (Correct answer)
- Triggers one micro-batch every second
- Enables continuous processing mode
Correct answer: Processes all available data then stops, like a batch job
availableNow=True causes the stream to process all currently available data in one or more micro-batches and then terminate.
Question 6: What is the purpose of the `ZORDER BY` command in Delta Lake?
- Sorts data alphabetically for human readability
- Co-locates related data in the same files to improve query filter performance (Correct answer)
- Compresses files using Z-standard compression
- Partitions data by a timestamp column
Correct answer: Co-locates related data in the same files to improve query filter performance
ZORDER BY uses a space-filling curve to co-locate rows with similar column values in the same files, improving data skipping for filtered queries.
Question 7: In Apache Spark, a 'wide transformation' differs from a 'narrow transformation' because it:
- Operates on multiple columns at once
- Requires a shuffle of data across partitions (Correct answer)
- Uses more CPU per task
- Can only be applied to RDDs, not DataFrames
Correct answer: Requires a shuffle of data across partitions
Wide transformations like groupBy and join require data to be exchanged (shuffled) between partitions, creating stage boundaries in the DAG.
LFC Apache Spark Data Engineering 2
Which Spark component is responsible for translating DataFrame operations into an optimized physical execution plan?