Data Processing Cycle 4 — Questions and Answers
Question 1: In stream processing, a 'tumbling window' cycle processes events in:
- Overlapping fixed-size time windows
- Non-overlapping fixed-size time windows (Correct answer)
- Variable-size windows based on event count
- Windows that slide forward by one event at a time
Correct answer: Non-overlapping fixed-size time windows
Tumbling windows are fixed-size and non-overlapping; each event belongs to exactly one window, making them simple for aggregation cycles.
Question 2: A 'sliding window' in a streaming data cycle differs from a tumbling window because it:
- Never overlaps adjacent windows
- Advances by a step smaller than its size, so consecutive windows share events (Correct answer)
- Processes events only when the window is completely full
- Resets after every single event
Correct answer: Advances by a step smaller than its size, so consecutive windows share events
Sliding windows move forward by a step interval smaller than the window size, so events can appear in multiple overlapping windows.
Question 3: Data lineage tracking in a processing cycle is primarily used to:
- Speed up query execution by caching results
- Trace the origin, transformations, and movement of data through the pipeline (Correct answer)
- Encrypt sensitive fields before loading
- Balance load across processing nodes
Correct answer: Trace the origin, transformations, and movement of data through the pipeline
Data lineage records where data came from and every transformation it underwent, enabling impact analysis, debugging, and compliance audits.
Question 4: In a MapReduce cycle, the 'shuffle and sort' phase occurs between:
- Input split and the Map phase
- The Map phase and the Reduce phase (Correct answer)
- The Reduce phase and output writing
- Two consecutive Map phases
Correct answer: The Map phase and the Reduce phase
Shuffle and sort transfers intermediate key-value pairs from mappers to the appropriate reducers, grouping all values for each key together.
Question 5: Which data processing cycle pattern is best suited for continuously updating fraud detection scores as transactions arrive?
- Weekly batch processing
- Monthly full refresh ETL
- Real-time stream processing (Correct answer)
- Yearly archive and purge cycle
Correct answer: Real-time stream processing
Real-time stream processing evaluates each transaction as it arrives, enabling immediate fraud detection with minimal latency.
Question 6: In the data processing lifecycle, 'data profiling' is best performed:
- After data is loaded into the production data warehouse
- Before transformation, to understand source data quality and structure (Correct answer)
- Only when end users report data errors
- During the archival phase to document retired datasets
Correct answer: Before transformation, to understand source data quality and structure
Profiling source data early reveals nulls, outliers, and format issues so transformation rules can be designed to handle them correctly.
Question 7: A 'dead letter queue' in a data pipeline cycle stores records that:
- Have been successfully processed and archived
- Failed processing and could not be handled by normal error logic (Correct answer)
- Are waiting to be picked up by the next scheduled batch
- Have been deduplicated and discarded
Correct answer: Failed processing and could not be handled by normal error logic
A dead letter queue isolates problematic records that repeatedly fail processing so they can be inspected and reprocessed without blocking the main pipeline.
Data Processing Cycle 4
In stream processing, a 'tumbling window' cycle processes events in: