Data Processing Data Processing Tools and Technologies 3

Question 1

What is a data pipeline?

Accepted Answer

A series of automated steps that move and transform data from source to destination

Answer

A data pipeline is a sequence of automated processes that ingest, transform, and move data from one or more sources to a destination for storage or analysis.

Question 2

What does idempotency mean in data processing?

Accepted Answer

Running the same processing job multiple times produces the same result without side effects

Answer

An idempotent pipeline can be run multiple times safely — re-running after a failure produces the same result without duplicating or corrupting data.

Question 3

What is a DAG in pipeline orchestration tools like Apache Airflow?

Accepted Answer

Directed Acyclic Graph — a structure defining task dependencies with no cycles

Answer

A DAG (Directed Acyclic Graph) defines pipeline tasks and their dependencies, ensuring tasks run in the correct order without circular dependencies.

Question 4

What is data serialization?

Accepted Answer

Converting data structures into a format that can be stored or transmitted and later reconstructed

Answer

Serialization converts in-memory data objects into a storable or transmittable format (like JSON, Avro, or Parquet) that can later be deserialized back into the original structure.

Question 5

What is the purpose of schema registry in streaming data architectures?

Accepted Answer

Centrally managing and enforcing data schemas for messages in a streaming system

Answer

A schema registry stores and enforces the schema for messages in a streaming pipeline, ensuring producers and consumers agree on the data structure and preventing breaking changes.

Data Processing Practice Test

Data Processing Practice Test

Data Processing Data Processing Tools and Technologies 3