LFC ETL & Data Integration 5

Question 1

A Gold-layer table needs to aggregate Silver data daily. Which scheduling pattern in Databricks Workflows best fits this batch requirement?

Accepted Answer

A cron-scheduled Job with a Delta table as source

Answer

Continuous streaming with 1-second trigger

Answer

Manual notebook execution only

Answer

Event-driven trigger on every Bronze file arrival

Question 2

When integrating with an external REST API as a data source in an ETL pipeline, what is the recommended pattern to avoid reprocessing all records on each run?

Accepted Answer

Use cursor-based or timestamp-based pagination to fetch only new records

Answer

Fetch all records on every run and deduplicate in the target

Answer

Cache the API response in Spark driver memory indefinitely

Answer

Use a broadcast join against the full API dataset

Question 3

What does 'data lineage' mean in the context of a lakehouse ETL pipeline?

Accepted Answer

The record of how data flows and transforms from source to destination tables

Answer

The order in which Spark tasks execute on a cluster

Answer

The physical storage path of Parquet files in cloud storage

Answer

The primary key constraint chain across related tables

Question 4

In Delta Live Tables, which expectation action causes the entire pipeline update to fail if a data quality constraint is violated?

Accepted Answer

ON VIOLATION FAIL UPDATE

Answer

ON VIOLATION DROP ROW

Answer

ON VIOLATION WARN

Answer

ON VIOLATION QUARANTINE

Question 5

Which Spark join strategy should be used when joining a large fact table with a small dimension table in an ETL pipeline to avoid shuffle?

Accepted Answer

Broadcast hash join

Answer

Sort-merge join

Answer

Skewed join with salting

Answer

Cross join

Question 6

What is the effect of calling VACUUM on a Delta Lake table after a large ETL load?

Accepted Answer

Removes old Parquet files that are no longer referenced by the transaction log

Answer

Recomputes all statistics and Z-Order indexes

Answer

Compacts small files into larger ones

Answer

Clears the Spark SQL result cache

Question 7

When designing an idempotent ETL pipeline stage, which technique ensures that re-running the pipeline does not produce duplicate records in the target Delta table?

Accepted Answer

Using MERGE with a unique business key to upsert records

Answer

Using INSERT INTO without a key check

Answer

Appending with UNION ALL and filtering duplicates in the UI

Answer

Setting spark.sql.shuffle.partitions=1 to serialize writes