LFC ETL & Data Integration 5

Question 1

A Gold-layer table needs to aggregate Silver data daily. Which scheduling pattern in Databricks Workflows best fits this batch requirement?

Accepted Answer

A cron-scheduled Job with a Delta table as source

Answer

A cron-scheduled Databricks Job reads from the Silver Delta table and writes aggregated results to the Gold table on a defined daily schedule.

Question 2

When integrating with an external REST API as a data source in an ETL pipeline, what is the recommended pattern to avoid reprocessing all records on each run?

Accepted Answer

Use cursor-based or timestamp-based pagination to fetch only new records

Answer

Cursor-based or timestamp-based incremental API calls retrieve only records created or updated since the last successful run, minimizing API calls and data volume.

Question 3

What does 'data lineage' mean in the context of a lakehouse ETL pipeline?

Accepted Answer

The record of how data flows and transforms from source to destination tables

Answer

Data lineage tracks the origin, movement, and transformation of data through the pipeline, enabling impact analysis, debugging, and compliance auditing.

Question 4

In Delta Live Tables, which expectation action causes the entire pipeline update to fail if a data quality constraint is violated?

Accepted Answer

ON VIOLATION FAIL UPDATE

Answer

ON VIOLATION FAIL UPDATE stops the DLT pipeline update and raises an error when any record violates the defined constraint, ensuring bad data never reaches downstream tables.

Question 5

Which Spark join strategy should be used when joining a large fact table with a small dimension table in an ETL pipeline to avoid shuffle?

Accepted Answer

Broadcast hash join

Answer

A broadcast hash join replicates the small dimension table to every executor, eliminating the expensive shuffle of the large fact table across the network.

(LFC) Lakehouse Fundamentals Certification Practice Test

(LFC) Lakehouse Fundamentals Certification Practice Test

LFC ETL & Data Integration 5