Azure Data Processing and Pipelines Flashcards

In Azure Databricks, what is a Delta Lake table?

Answer: An ACID-compliant storage layer built on Parquet files with a transaction log

Delta Lake adds ACID transactions, schema enforcement, and time travel to Parquet files by maintaining a JSON transaction log alongside the data.

What Azure Databricks feature allows you to query the state of a Delta table at a previous point in time?

Answer: Time travel

Delta Lake time travel allows querying historical table versions using VERSION AS OF or TIMESTAMP AS OF syntax.

In Azure Databricks, which command is used to optimize a Delta table by compacting small files into larger ones?

Answer: OPTIMIZE

The OPTIMIZE command in Delta Lake compacts small Parquet files into larger ones, improving query performance by reducing file overhead.

What does the VACUUM command do on a Delta Lake table?

Answer: Removes old data files no longer referenced by the transaction log

VACUUM deletes data files older than the retention threshold that are no longer referenced by the Delta transaction log, reclaiming storage.

Which Azure service is used to ingest high-velocity streaming data such as telemetry from millions of IoT devices before processing in Azure Databricks?

Answer: Azure Event Hubs

Azure Event Hubs is a massively scalable event streaming platform that can ingest millions of events per second for downstream processing.

What is the primary benefit of using Azure Data Factory's data flow (mapping data flow) over a custom Spark script?

Answer: Codeless, visually designed transformations that run on Spark clusters

Mapping data flows provide a visual, codeless interface for building Spark transformations, making complex ETL accessible without Spark expertise.