LFC Apache Spark Data Engineering 3

Question 1

Which Spark API abstraction represents an immutable, distributed collection of objects with no schema enforcement?

Accepted Answer

RDD

Answer

RDD (Resilient Distributed Dataset) is the lowest-level Spark abstraction — distributed and fault-tolerant but without schema or query optimization.

Question 2

In Delta Lake's transaction log, what file type records each committed transaction's metadata and statistics?

Accepted Answer

JSON entry in the _delta_log directory

Answer

Every Delta Lake commit writes a JSON file to the _delta_log directory recording the operations, file adds/removes, and column statistics.

Question 3

What Spark technique reduces data movement by pushing filter conditions down to the data source before loading into memory?

Accepted Answer

Predicate pushdown

Answer

Predicate pushdown allows Spark to pass filter conditions to the storage layer (e.g., Parquet or Delta) so only matching rows are read.

Question 4

When should you use `df.cache()` versus `df.persist(StorageLevel.DISK_ONLY)` in Spark?

Accepted Answer

cache() stores in memory only; persist() allows specifying storage level including disk

Answer

cache() is shorthand for persist(MEMORY_AND_DISK) in DataFrames, while persist() lets you specify storage levels like DISK_ONLY, MEMORY_ONLY, or OFF_HEAP.

Question 5

In a Databricks lakehouse, what is the role of the Unity Catalog?

Accepted Answer

A centralized governance layer for data, AI assets, and fine-grained access control across workspaces

Answer

Unity Catalog provides centralized governance, lineage tracking, and fine-grained access control for tables, volumes, and ML models across multiple Databricks workspaces.

(LFC) Lakehouse Fundamentals Certification Practice Test

(LFC) Lakehouse Fundamentals Certification Practice Test

LFC Apache Spark Data Engineering 3