ETL Testing ETL Testing Tools and Automation 1 — Questions and Answers
Question 1: Which open-source framework is commonly used in the US for automated ETL pipeline testing with SQL-based assertions?
- Selenium WebDriver
- dbt (data build tool) with dbt test (Correct answer)
- JMeter
- Postman
Correct answer: dbt (data build tool) with dbt test
dbt's built-in test framework allows data engineers to write SQL-based assertions (not_null, unique, relationships) that run automatically after each transformation model.
Question 2: What is the primary purpose of Apache Griffin in the ETL testing ecosystem?
- Orchestrating ETL job schedules
- Providing a data quality measurement platform for batch and streaming pipelines (Correct answer)
- Generating test data for ETL pipelines
- Monitoring ETL server CPU usage
Correct answer: Providing a data quality measurement platform for batch and streaming pipelines
Apache Griffin is an open-source data quality service that measures accuracy, completeness, timeliness, and other quality dimensions across ETL pipelines at scale.
Question 3: In ETL automation testing, what is a 'data diff' tool used for?
- Comparing source and target datasets row-by-row to identify discrepancies automatically (Correct answer)
- Diffing ETL job configuration files
- Measuring load time differences between runs
- Comparing server hardware specifications
Correct answer: Comparing source and target datasets row-by-row to identify discrepancies automatically
Data diff tools automatically compare datasets at the row and column level to surface mismatches between source, staging, and target without manual inspection.
Question 4: Which AWS service is commonly tested in US-based ETL pipelines to verify that Glue jobs correctly transform data?
- AWS Lambda test harness
- AWS Glue job with PyTest and GlueContext mocking (Correct answer)
- AWS CloudWatch alarms only
- AWS S3 versioning
Correct answer: AWS Glue job with PyTest and GlueContext mocking
AWS Glue ETL jobs are typically tested using PyTest with mocked GlueContext or local Glue development endpoints to validate transformation logic before production deployment.
Question 5: What does 'data lineage tracking' provide that is valuable for ETL testing?
- Real-time monitoring of server disk usage
- A visual or programmatic map of how data flows from source to target, enabling impact analysis and root cause diagnosis (Correct answer)
- A list of all ETL job execution logs
- An audit of user access to the data warehouse
Correct answer: A visual or programmatic map of how data flows from source to target, enabling impact analysis and root cause diagnosis
Data lineage tools show the full path data travels through an ETL pipeline, helping testers quickly identify which transformation or load step introduced a defect.
Question 6: Which ETL testing approach uses a known 'golden dataset' with pre-validated expected outputs to automatically verify transformations?
- Exploratory testing
- Golden dataset regression testing (Correct answer)
- Ad-hoc querying
- Smoke testing
Correct answer: Golden dataset regression testing
Golden dataset regression testing runs a curated, pre-verified input dataset through the ETL pipeline and automatically compares the output against known-correct expected results.
ETL Testing ETL Testing Tools and Automation 1
Which open-source framework is commonly used in the US for automated ETL pipeline testing with SQL-based assertions?