Data Engineering Data Quality and Testing 2 — Questions and Answers
Question 1: What is 'data lineage' and why is it important for data quality?
- The history of data ownership changes
- The end-to-end tracking of data from origin through all transformations to its final destination (Correct answer)
- The process of archiving old data versions
- The audit log of who accessed data
Correct answer: The end-to-end tracking of data from origin through all transformations to its final destination
Data lineage maps the journey of data from source to destination, enabling root cause analysis when data quality issues occur.
Question 2: Which approach to data quality testing runs assertions directly in SQL against your data warehouse tables?
- Unit testing with mocked data
- In-warehouse testing (e.g., dbt tests) (Correct answer)
- Schema registry validation
- API contract testing
Correct answer: In-warehouse testing (e.g., dbt tests)
In-warehouse testing (used by dbt and similar tools) compiles assertions into SQL queries that run directly against the target tables.
Question 3: What is 'referential integrity' in the context of data warehouse quality?
- Ensuring all fact table foreign keys have matching dimension table records (Correct answer)
- Verifying that column names match their documented definitions
- Confirming that numeric measures sum to expected totals
- Checking that all records have populated primary keys
Correct answer: Ensuring all fact table foreign keys have matching dimension table records
Referential integrity ensures every foreign key in the fact table points to an existing record in the corresponding dimension table.
Question 4: What is an 'anomaly detection' test in data quality pipelines?
- A test that catches duplicate records
- A test that flags unexpected deviations from historical data patterns (Correct answer)
- A test that validates schema changes between pipeline runs
- A test that checks referential integrity constraints
Correct answer: A test that flags unexpected deviations from historical data patterns
Anomaly detection tests compare current data metrics against historical baselines to flag unusual volumes, distributions, or values.
Question 5: In dbt, what is the difference between a 'test' and a 'snapshot'?
- Tests validate data quality assertions; snapshots capture slowly changing dimension history (Correct answer)
- Tests run in staging; snapshots run in production
- Tests are written in Python; snapshots are written in SQL
- Tests check schema; snapshots check data values
Correct answer: Tests validate data quality assertions; snapshots capture slowly changing dimension history
dbt tests assert data quality conditions (like uniqueness or not-null), while snapshots implement SCD Type 2 by capturing row changes over time.
Question 6: What does 'data profiling' accomplish in a data engineering workflow?
- Optimizes query execution plans for faster performance
- Analyzes dataset characteristics like cardinality, null rates, and distributions to understand data quality (Correct answer)
- Encrypts sensitive data fields for compliance
- Partitions large tables to reduce scan costs
Correct answer: Analyzes dataset characteristics like cardinality, null rates, and distributions to understand data quality
Data profiling examines datasets to produce statistical summaries (null counts, distinct values, min/max) that reveal data quality issues before transformation.
Data Engineering Data Quality and Testing 2
What is 'data lineage' and why is it important for data quality?