Data Engineering ETL and ELT Pipelines 3 — Questions and Answers
Question 1: Which orchestration tool models pipelines as Directed Acyclic Graphs (DAGs) of tasks?
- Apache Airflow (Correct answer)
- Apache Kafka
- Redis
- Nginx
Correct answer: Apache Airflow
Airflow defines workflows as DAGs where each node is a task with explicit dependencies.
Question 2: A pipeline must process events within seconds of arrival. Which paradigm fits best?
- Streaming (real-time) processing (Correct answer)
- Nightly batch processing
- Manual CSV exports
- Quarterly full reloads
Correct answer: Streaming (real-time) processing
Streaming processes records continuously as they arrive, achieving low end-to-end latency.
Question 3: In dimensional modeling, a Type 2 Slowly Changing Dimension handles updates by what method?
- Adding a new row to preserve historical versions (Correct answer)
- Overwriting the existing value in place
- Deleting the old record entirely
- Storing changes only in logs
Correct answer: Adding a new row to preserve historical versions
SCD Type 2 inserts a new row with validity dates, keeping the full history of the attribute.
Question 4: What is the main risk of using SELECT * during extraction from a wide source table?
- Pulling unnecessary columns and breaking when schema changes (Correct answer)
- Guaranteed data corruption
- Inability to filter rows
- Automatic deduplication of records
Correct answer: Pulling unnecessary columns and breaking when schema changes
SELECT * transfers all columns wastefully and makes the pipeline fragile to upstream schema changes.
Question 5: Which storage layer typically holds raw, unprocessed data in a modern data lakehouse pattern?
- The bronze (raw) layer (Correct answer)
- The gold (curated) layer
- The presentation layer
- The semantic layer
Correct answer: The bronze (raw) layer
The medallion architecture's bronze layer stores raw ingested data before refinement.
Question 6: A dead-letter queue in a pipeline is used to do what?
- Capture records that fail processing for later inspection (Correct answer)
- Speed up successful record throughput
- Store backup credentials
- Compress the output files
Correct answer: Capture records that fail processing for later inspection
A dead-letter queue isolates malformed or failed messages so the main flow continues uninterrupted.
Question 7: Which practice helps ensure data quality is verified within the pipeline itself?
- Automated data validation tests on row counts and null thresholds (Correct answer)
- Manually eyeballing dashboards weekly
- Disabling logging to save space
- Skipping checks to reduce runtime
Correct answer: Automated data validation tests on row counts and null thresholds
Embedding automated assertions (row counts, nulls, ranges) catches quality issues before data reaches users.
Data Engineering ETL and ELT Pipelines 3
Which orchestration tool models pipelines as Directed Acyclic Graphs (DAGs) of tasks?