Data Engineering Orchestrating Data Workflows 3 — Questions and Answers
Question 1: What is the main advantage of defining workflows as code (e.g., Python DAGs) over a GUI-only tool?
- It removes the need for testing
- Version control, code review, and reproducibility (Correct answer)
- It eliminates all runtime errors
- It requires no scheduler
Correct answer: Version control, code review, and reproducibility
Workflows-as-code can be versioned, reviewed, and tested like any other software.
Question 2: In Airflow, what does the schedule_interval '@daily' do?
- Runs the DAG once at midnight each day (Correct answer)
- Runs every hour
- Runs only when triggered manually
- Runs continuously
Correct answer: Runs the DAG once at midnight each day
'@daily' schedules the DAG to run once per day at midnight.
Question 3: Which retry strategy helps avoid overwhelming a downstream system after repeated failures?
- Immediate constant retries
- Exponential backoff (Correct answer)
- Disabling retries entirely
- Parallel duplicate retries
Correct answer: Exponential backoff
Exponential backoff increases the delay between retries, reducing load on failing systems.
Question 4: What problem does a task dependency graph primarily prevent?
- Running tasks before their inputs are ready (Correct answer)
- High storage costs
- Slow network bandwidth
- Schema drift
Correct answer: Running tasks before their inputs are ready
Dependency graphs ensure a task only runs after its upstream prerequisites complete.
Question 5: In Prefect or Dagster, what is a key advantage over a pure cron schedule?
- No code is needed
- Built-in retries, observability, and dependency management (Correct answer)
- It runs without any compute
- It only supports one task per flow
Correct answer: Built-in retries, observability, and dependency management
Modern orchestrators add observability, retries, and dependency handling that cron lacks.
Question 6: What does 'idempotency key' commonly guard against in pipeline writes?
- Slow queries
- Duplicate records from re-processing the same batch (Correct answer)
- Missing schemas
- Unencrypted data
Correct answer: Duplicate records from re-processing the same batch
An idempotency key lets the system detect and skip duplicate writes during retries.
Question 7: Which scenario is best handled by event-driven orchestration rather than time-based scheduling?
- A monthly financial report
- Triggering a pipeline the moment a new file lands in cloud storage (Correct answer)
- A fixed nightly batch load
- A weekly cleanup job
Correct answer: Triggering a pipeline the moment a new file lands in cloud storage
Event-driven orchestration reacts to events like file arrival rather than waiting for a clock.
Data Engineering Orchestrating Data Workflows 3
What is the main advantage of defining workflows as code (e.g., Python DAGs) over a GUI-only tool?