Data Engineering Real-Time Streaming Architectures Questions and Answers

Question 1

A financial services company wants to monitor credit card transactions for fraudulent activity. The system must group all transactions by a specific user that occur closely together, but the time between these bursts of activity is unpredictable. A new group should start only after a significant period of user inactivity. Which windowing strategy is most suitable for this scenario?

Accepted Answer

Session Windows

Answer

Session windows are designed specifically for this use case. They group events based on periods of activity, which are terminated by a predefined gap of inactivity (a timeout). This allows the system to dynamically create windows for each user's transaction burst without having fixed start or end times.

Question 2

In a real-time streaming architecture that processes data using event time, what is the primary function of a watermark?

Accepted Answer

To signal to the processing engine a point in time beyond which it is unlikely to receive older events, allowing it to close windows and trigger computations.

Answer

A watermark is a mechanism that tracks the progress of event time in a stream. It provides a heuristic for the system to know when it has likely received all the events for a particular time window, enabling it to finalize calculations for that window even if some data arrives out of order or late. This balances the need for accuracy with the need to produce timely results and manage state.

Question 3

A data engineering team is building a real-time dashboard to display the number of unique visitors on their website over the last 15 minutes. Which type of stream processing is fundamentally required to implement this feature?

Accepted Answer

Stateful processing

Answer

Calculating the number of unique visitors requires the system to remember which visitor IDs it has already seen within the current time window. This memory of past events is known as 'state'. Therefore, this is a stateful processing operation. A stateless operation processes each event independently without knowledge of previous events.

Question 4

Which of the following is a defining characteristic of the Kappa Architecture compared to the Lambda Architecture?

Accepted Answer

It simplifies the architecture by using a single stream-processing engine to handle both real-time queries and historical data reprocessing from an append-only log.

Answer

The Kappa Architecture was proposed as a simplification of the Lambda Architecture. Its core principle is to eliminate the batch layer and use a single, unified stream processing pipeline for all tasks. Historical analysis is achieved by replaying the data from a canonical, immutable log (like Apache Kafka) through the same streaming engine.

Question 5

In a streaming data pipeline, a processing stage is consistently unable to keep up with the rate of data it receives from an upstream stage. This leads to growing memory usage, increased latency, and potential system instability. What is the term for the mechanism designed to mitigate this specific problem?

Accepted Answer

Backpressure

Answer

Backpressure is a flow control mechanism where a slower downstream consumer can signal to a faster upstream producer to reduce the rate of data transfer. This prevents the consumer from being overwhelmed, which could lead to buffer overflows, data loss, or system failure.

Data Engineering Practice Test

Data Engineering Practice Test

Data Engineering Real-Time Streaming Architectures Questions and Answers