LFC Apache Spark Data Engineering 2

Question 1

Which Spark component is responsible for translating DataFrame operations into an optimized physical execution plan?

Accepted Answer

Catalyst Optimizer

Answer

The Catalyst Optimizer analyzes logical plans and applies rule-based and cost-based optimizations to generate an efficient physical plan.

Question 2

In Delta Lake, what mechanism prevents two concurrent write operations from corrupting the same table?

Accepted Answer

Optimistic concurrency control

Answer

Delta Lake uses optimistic concurrency control, detecting conflicts at commit time and retrying or failing conflicting transactions.

Question 3

What does the `spark.sql.shuffle.partitions` configuration control?

Accepted Answer

Number of partitions after a wide transformation like join or aggregation

Answer

This setting determines how many partitions Spark creates during shuffle operations such as groupBy, join, and distinct.

Question 4

Which file format used in lakehouses stores data in columnar format with built-in schema and statistics per row group?

Accepted Answer

Parquet

Answer

Parquet stores data columnar with per-column statistics per row group, enabling predicate pushdown and efficient compression.

Question 5

When using Spark Structured Streaming, what does the `trigger(availableNow=True)` option do?

Accepted Answer

Processes all available data then stops, like a batch job

Answer

availableNow=True causes the stream to process all currently available data in one or more micro-batches and then terminate.

(LFC) Lakehouse Fundamentals Certification Practice Test