Data Processing Cycle 3 — Questions and Answers
Question 1: In a data quality lifecycle, which cycle step validates that transformed data conforms to expected business rules before loading?
- Data profiling during extraction
- Pre-load validation / data quality checks (Correct answer)
- Post-load reconciliation only
- Schema inference at query time
Correct answer: Pre-load validation / data quality checks
Pre-load validation applies business rule checks on transformed data to catch errors before they reach the target system.
Question 2: A circular dependency in a data pipeline cycle means:
- Two pipelines share the same source system
- Task A depends on Task B, and Task B depends on Task A, creating an infinite loop (Correct answer)
- Data flows in a round-robin pattern across nodes
- The same dataset is processed twice per cycle
Correct answer: Task A depends on Task B, and Task B depends on Task A, creating an infinite loop
A circular dependency creates a deadlock where neither task can start because each waits for the other to complete first.
Question 3: In data processing, 'idempotency' means a pipeline step can be run multiple times in the same cycle and:
- Produce faster results each subsequent run
- Produce the same result without causing duplicate or incorrect data (Correct answer)
- Automatically scale to handle more records
- Skip previously processed records entirely
Correct answer: Produce the same result without causing duplicate or incorrect data
An idempotent operation produces the same output regardless of how many times it is executed, preventing data corruption during retries.
Question 4: Which scheduling strategy triggers a new processing cycle only when the upstream data source signals that new data is available?
- Time-based (cron) scheduling
- Event-driven scheduling (Correct answer)
- Manual trigger scheduling
- Round-robin scheduling
Correct answer: Event-driven scheduling
Event-driven scheduling starts a processing cycle in response to a trigger event, such as a file arrival or message queue notification.
Question 5: The 'medallion architecture' in data lakes uses three cycle stages. In order, these are:
- Gold → Silver → Bronze
- Bronze → Silver → Gold (Correct answer)
- Raw → Curated → Aggregated → Reporting
- Ingest → Process → Archive
Correct answer: Bronze → Silver → Gold
Bronze holds raw ingested data, Silver holds cleaned/conformed data, and Gold holds business-level aggregated data ready for analytics.
Question 6: When a data processing cycle fails midway, a 'checkpoint' mechanism allows the job to:
- Restart from the beginning with a clean state
- Resume processing from the last successfully completed checkpoint rather than from scratch (Correct answer)
- Discard all in-flight records and wait for the next scheduled run
- Alert operators and pause indefinitely
Correct answer: Resume processing from the last successfully completed checkpoint rather than from scratch
Checkpoints save progress state so that a restarted job can pick up where it left off, avoiding expensive full reprocessing.
Question 7: In a data warehouse load cycle, a 'full refresh' strategy differs from an 'incremental' strategy in that full refresh:
- Only loads records changed since the last run
- Truncates and reloads the entire target table every cycle (Correct answer)
- Uses CDC logs to capture changes
- Merges new records with existing ones using a surrogate key
Correct answer: Truncates and reloads the entire target table every cycle
A full refresh drops and reloads all data every cycle, ensuring complete accuracy at the cost of higher processing time and resource usage.
Data Processing Cycle 3
In a data quality lifecycle, which cycle step validates that transformed data conforms to expected business rules before loading?