Medallion Architecture Layers Flashcards

What is the recommended approach when a source system changes its schema and sends new fields to the Bronze layer?

Answer: Allow schema evolution so Bronze absorbs the new fields without pipeline failure

Bronze should support schema evolution (e.g., Delta Lake's schema evolution) to handle upstream changes without breaking ingestion.

A financial services firm must prove that a Gold-layer aggregated metric can be traced back to specific source transactions. Which Medallion feature enables this?

Answer: Data lineage through retained Bronze and Silver layers

Because Bronze and Silver are retained, auditors can trace a Gold metric back through Silver cleansing to the original Bronze raw records.

Which Medallion layer is MOST appropriate for storing a feature engineering table used as input to a machine learning model?

Answer: Silver, if features are computed from cleansed data without business aggregation

Feature tables derived from clean, conformed records but without business-level aggregation are typically housed in the Silver layer.

When implementing incremental processing in Medallion Architecture, which Delta Lake feature is used to efficiently identify new Bronze records to process into Silver?

Answer: Delta Change Data Feed (CDF) or DESCRIBE HISTORY

Delta Change Data Feed (CDF) exposes row-level changes, and DESCRIBE HISTORY tracks versions, both enabling incremental Silver processing.

A Bronze table is growing rapidly and query performance is degrading. Which operation should be applied WITHOUT moving data out of the Bronze layer?

Answer: Running OPTIMIZE and Z-ORDER on the Bronze Delta table

OPTIMIZE compacts small files and Z-ORDER co-locates related data, improving scan performance on Delta tables within any layer.

Which Medallion layer would you query if you need to reconstruct the state of a record as it appeared at a specific historical point in time?

Answer: Bronze, using Delta time travel to the ingestion timestamp

Bronze retains the original raw records; Delta time travel allows querying Bronze at a specific version or timestamp for historical reconstruction.

In a streaming Medallion pipeline, data flows from Kafka into Bronze continuously. Which Spark API is typically used for this ingestion pattern?

Answer: Structured Streaming with Auto Loader or readStream

Structured Streaming (with Auto Loader for file sources or readStream for Kafka) enables continuous, low-latency ingestion into Bronze Delta tables.