Orchestrating Data Workflows Flashcards

In Apache Airflow, what does a DAG represent?

Answer: A directed acyclic graph defining task dependencies

A DAG (Directed Acyclic Graph) defines tasks and the order in which they run, with no cycles.

Which Airflow concept controls how many task instances can run concurrently for a single DAG?

Answer: concurrency (max_active_tasks)

The concurrency (max_active_tasks) setting limits how many task instances run at once within a DAG.

What is the purpose of an idempotent task in a data pipeline?

Answer: Re-running it produces the same result without side effects

Idempotent tasks can be safely retried because repeated execution yields the same end state.

In a workflow scheduler, what is 'backfilling'?

Answer: Running a DAG for past dates that were missed or newly added

Backfilling executes a pipeline over historical intervals to populate missing data.

Which statement best describes the difference between scheduling and orchestration?

Answer: Scheduling triggers jobs by time; orchestration manages dependencies and coordination

Scheduling decides when jobs run, while orchestration coordinates dependencies, retries, and data flow across tasks.

In Airflow, what does an XCom enable?

Answer: Passing small amounts of data between tasks

XComs (cross-communications) let tasks exchange small pieces of data within a DAG.

What is a common reason to use a sensor in an orchestration tool?

Answer: To wait for an external condition like a file or partition to appear

Sensors pause a workflow until a specified external event or condition is met.