LFC ETL & Data Integration 3

Question 1

What is the role of the Bronze layer in a Medallion architecture ETL pipeline?

Accepted Answer

Stores raw, unmodified ingested data

Answer

The Bronze layer preserves raw source data as-is, providing a historical record and enabling reprocessing if downstream logic changes.

Question 2

When using Auto Loader in Databricks, what service does it rely on to efficiently detect new files in cloud storage without listing all files?

Accepted Answer

Cloud object store notification events (e.g., S3 Event Notifications)

Answer

Auto Loader uses cloud provider file notification services (like AWS S3 Event Notifications or Azure Event Grid) to detect new files incrementally without costly directory listings.

Question 3

Which DLT keyword is used to define a table that performs data quality checks and quarantines bad records?

Accepted Answer

CONSTRAINT ... ON VIOLATION DROP ROW

Answer

DLT's CONSTRAINT clause with ON VIOLATION DROP ROW (or FAIL UPDATE) enforces data quality rules and removes or fails on records that violate them.

Question 4

In Spark structured streaming, what does 'exactly-once' semantics require from both the source and sink?

Accepted Answer

Idempotent writes and replayable sources

Answer

Exactly-once requires sources to be replayable (e.g., Kafka with offsets) and sinks to support idempotent writes so reprocessed data doesn't cause duplicates.

Question 5

A Silver-layer Delta table should merge CDC records from Bronze. Which MERGE operation correctly handles deletes from the source?

Accepted Answer

WHEN MATCHED AND src.op='D' THEN DELETE

Answer

To handle CDC deletes, you add a WHEN MATCHED AND src.op='D' THEN DELETE clause so rows flagged as deleted in the source are removed from the target.

(LFC) Lakehouse Fundamentals Certification Practice Test

(LFC) Lakehouse Fundamentals Certification Practice Test

LFC ETL & Data Integration 3