LFC ETL & Data Integration 3 — Questions and Answers
Question 1: What is the role of the Bronze layer in a Medallion architecture ETL pipeline?
- Stores curated, business-ready aggregated data
- Stores raw, unmodified ingested data (Correct answer)
- Stores cleansed data with applied business rules
- Stores only streaming data from IoT devices
Correct answer: Stores raw, unmodified ingested data
The Bronze layer preserves raw source data as-is, providing a historical record and enabling reprocessing if downstream logic changes.
Question 2: When using Auto Loader in Databricks, what service does it rely on to efficiently detect new files in cloud storage without listing all files?
- Apache Kafka triggers
- Cloud object store notification events (e.g., S3 Event Notifications) (Correct answer)
- Delta table VACUUM events
- Databricks Jobs scheduler polling
Correct answer: Cloud object store notification events (e.g., S3 Event Notifications)
Auto Loader uses cloud provider file notification services (like AWS S3 Event Notifications or Azure Event Grid) to detect new files incrementally without costly directory listings.
Question 3: Which DLT keyword is used to define a table that performs data quality checks and quarantines bad records?
- CONSTRAINT ... ON VIOLATION DROP ROW (Correct answer)
- ENFORCE SCHEMA
- QUARANTINE TABLE
- STREAMING LIVE TABLE
Correct answer: CONSTRAINT ... ON VIOLATION DROP ROW
DLT's CONSTRAINT clause with ON VIOLATION DROP ROW (or FAIL UPDATE) enforces data quality rules and removes or fails on records that violate them.
Question 4: In Spark structured streaming, what does 'exactly-once' semantics require from both the source and sink?
- Idempotent writes and replayable sources (Correct answer)
- At-least-once delivery and duplicate filtering only in the UI
- Only a unique primary key on the sink table
- A stateless transformation graph
Correct answer: Idempotent writes and replayable sources
Exactly-once requires sources to be replayable (e.g., Kafka with offsets) and sinks to support idempotent writes so reprocessed data doesn't cause duplicates.
Question 5: A Silver-layer Delta table should merge CDC records from Bronze. Which MERGE operation correctly handles deletes from the source?
- WHEN NOT MATCHED THEN INSERT
- WHEN MATCHED AND src.op='D' THEN DELETE (Correct answer)
- WHEN MATCHED THEN UPDATE SET *
- WHEN NOT MATCHED BY SOURCE THEN UPDATE
Correct answer: WHEN MATCHED AND src.op='D' THEN DELETE
To handle CDC deletes, you add a WHEN MATCHED AND src.op='D' THEN DELETE clause so rows flagged as deleted in the source are removed from the target.
Question 6: What does 'schema-on-read' mean in a lakehouse ETL context?
- The schema is enforced when data is written to storage
- The schema is applied and interpreted when data is queried, not at ingestion (Correct answer)
- Spark infers the schema from Hive Metastore at write time
- Only Avro files support schema-on-read in lakehouses
Correct answer: The schema is applied and interpreted when data is queried, not at ingestion
Schema-on-read means raw data is stored without strict schema enforcement, and the schema is applied dynamically when the data is read or queried.
Question 7: Which Databricks utility function is commonly used to incrementally process files from a directory using a watermark to track what has already been ingested?
- spark.read.format('delta').load()
- dbutils.fs.ls() with manual state tracking
- spark.readStream.format('cloudFiles') — Auto Loader (Correct answer)
- spark.sql('COPY INTO ...')
Correct answer: spark.readStream.format('cloudFiles') — Auto Loader
Auto Loader (cloudFiles format) automatically tracks which files have been processed using checkpointing, enabling reliable incremental ingestion.
LFC ETL & Data Integration 3
What is the role of the Bronze layer in a Medallion architecture ETL pipeline?