Data Processing Cycle 5 — Questions and Answers
Question 1: In data processing, 'backpressure' during a streaming cycle refers to:
- Increasing CPU clock speed to process faster
- A mechanism where a slow consumer signals upstream producers to slow down data emission (Correct answer)
- Compressing data before transmission to reduce bandwidth
- Replaying past events from a message queue
Correct answer: A mechanism where a slow consumer signals upstream producers to slow down data emission
Backpressure prevents a fast producer from overwhelming a slow consumer by propagating a slow-down signal upstream through the pipeline.
Question 2: The PDCA cycle (Plan-Do-Check-Act) is applied in data processing quality management to:
- Design database schema migrations
- Continuously improve data pipeline quality through iterative testing and adjustment (Correct answer)
- Schedule nightly ETL batch jobs
- Partition large datasets across storage nodes
Correct answer: Continuously improve data pipeline quality through iterative testing and adjustment
PDCA provides a repeating improvement loop: plan changes to data quality processes, implement them, measure outcomes, and adjust accordingly.
Question 3: In an ELT (Extract, Load, Transform) cycle compared to ETL, transformation occurs:
- Before data is moved out of the source system
- After raw data is loaded into the target data warehouse or lake (Correct answer)
- During the extraction step using source-side stored procedures
- In a separate staging server outside the data warehouse
Correct answer: After raw data is loaded into the target data warehouse or lake
ELT loads raw data first, then leverages the compute power of the target system (e.g., BigQuery or Snowflake) to run transformations.
Question 4: Which scenario best illustrates 'late-arriving data' in a streaming processing cycle?
- A sensor sends data ahead of the scheduled window
- An event with a timestamp from 10 minutes ago arrives after the processing window for that period has already closed (Correct answer)
- A batch job runs 10 minutes before its scheduled time
- A data file is delivered to the wrong storage bucket
Correct answer: An event with a timestamp from 10 minutes ago arrives after the processing window for that period has already closed
Late-arriving data has an event timestamp that falls within a past window that the system has already finalized, requiring special handling like watermarks.
Question 5: In a data warehouse, a 'Type 2 Slowly Changing Dimension' (SCD2) cycle handles attribute changes by:
- Overwriting the old value with the new value in-place
- Adding a new row with the updated value and marking the old row as expired (Correct answer)
- Deleting the old record and inserting only the new record
- Storing only the most recent change in a separate audit table
Correct answer: Adding a new row with the updated value and marking the old row as expired
SCD2 preserves full history by inserting a new dimension row for each change and closing the prior row with an end date or a current-row flag.
Question 6: A 'watermark' in stream processing is used to:
- Encrypt sensitive fields in the data stream
- Define a threshold in event time that tells the system when to finalize results for a given window (Correct answer)
- Mark records that have been processed as read-only
- Partition streaming data across parallel processing threads
Correct answer: Define a threshold in event time that tells the system when to finalize results for a given window
A watermark is a heuristic that estimates how far behind real-time the pipeline is willing to wait for late events before closing a window.
Question 7: In a data replication cycle, 'eventual consistency' means that:
- All replicas are updated synchronously before any read is served
- Given enough time without new updates, all replicas will converge to the same value (Correct answer)
- Data is never replicated more than once per cycle
- The primary node always holds the most up-to-date copy
Correct answer: Given enough time without new updates, all replicas will converge to the same value
Eventual consistency allows temporary divergence between replicas but guarantees they will synchronize given sufficient propagation time.
Data Processing Cycle 5
In data processing, 'backpressure' during a streaming cycle refers to: