CDP (Big Data Science) 5 — Questions and Answers
Question 1: In the context of big data governance, what is 'data lineage'?
- The process of classifying data by sensitivity for regulatory compliance
- The ability to trace the origin, movement, and transformation of data throughout its lifecycle (Correct answer)
- A metadata catalog that indexes all datasets stored in a data lake
- The versioning system used to track schema changes over time
Correct answer: The ability to trace the origin, movement, and transformation of data throughout its lifecycle
Data lineage tracks where data originated, how it has been transformed, and where it flows, enabling auditability and impact analysis.
Question 2: Which of the following best describes 'model drift' in deployed machine learning systems?
- The gradual degradation of model weights due to floating-point precision loss over time
- A decline in model prediction accuracy caused by changes in real-world data distributions after training (Correct answer)
- The process of incrementally retraining a model on new data batches
- The divergence between training loss and validation loss during model fitting
Correct answer: A decline in model prediction accuracy caused by changes in real-world data distributions after training
Model drift occurs when the statistical properties of the input data change over time, causing a model trained on historical data to make less accurate predictions.
Question 3: What is the primary advantage of using a columnar database over a row-oriented database for OLAP workloads?
- Columnar databases support faster single-record inserts and updates
- Columnar databases read only the columns needed for a query, reducing I/O for analytical aggregations (Correct answer)
- Columnar databases enforce ACID transactions more efficiently than row-oriented stores
- Columnar databases are better suited for high-concurrency transactional workloads
Correct answer: Columnar databases read only the columns needed for a query, reducing I/O for analytical aggregations
Columnar storage allows analytical queries to read only the relevant columns from disk, dramatically reducing I/O compared to reading entire rows.
Question 4: In Apache Spark, what is the significance of the 'DAG' (Directed Acyclic Graph) scheduler?
- It manages cluster resource allocation by tracking node CPU and memory availability
- It represents the logical execution plan of a Spark job, enabling optimization and fault recovery (Correct answer)
- It stores RDD lineage information as a serialized graph in HDFS for checkpointing
- It routes shuffle data between executors using a graph-based routing protocol
Correct answer: It represents the logical execution plan of a Spark job, enabling optimization and fault recovery
Spark's DAG scheduler translates user operations into a graph of stages and tasks, optimizing the execution plan and enabling recomputation of lost partitions from lineage.
Question 5: Which approach does the 'Kappa architecture' take compared to Lambda architecture?
- Kappa uses three layers — speed, batch, and serving — for improved fault tolerance
- Kappa eliminates the batch layer entirely, using only a replayable streaming layer for all processing (Correct answer)
- Kappa introduces a caching layer between the speed and serving layers to reduce latency
- Kappa requires separate clusters for historical reprocessing and real-time streaming
Correct answer: Kappa eliminates the batch layer entirely, using only a replayable streaming layer for all processing
Kappa architecture simplifies Lambda by using a single streaming system for both real-time and historical data processing, leveraging log replay capabilities.
Question 6: What is the purpose of 'checkpointing' in distributed stream processing frameworks like Flink or Spark Streaming?
- To compress accumulated stream state to reduce memory consumption during long-running jobs
- To periodically snapshot the state of a streaming job so it can be recovered after a failure (Correct answer)
- To validate that output records match expected values during integration testing
- To throttle input data rates when downstream consumers cannot keep pace
Correct answer: To periodically snapshot the state of a streaming job so it can be recovered after a failure
Checkpointing saves a consistent snapshot of operator state and stream positions, allowing the job to resume from the last checkpoint after a failure rather than restarting from scratch.
Question 7: In big data security, what does 'column-level encryption' provide that whole-database encryption does not?
- Faster encryption throughput by parallelizing encryption across individual columns
- Granular protection allowing sensitive columns to be encrypted independently while leaving others queryable in plaintext (Correct answer)
- The ability to encrypt data in transit between distributed nodes without key management overhead
- Automatic key rotation for encrypted columns without requiring application-level changes
Correct answer: Granular protection allowing sensitive columns to be encrypted independently while leaving others queryable in plaintext
Column-level encryption protects specific sensitive fields (e.g., SSN, credit card numbers) independently, allowing non-sensitive columns to remain accessible without decryption overhead.
CDP (Big Data Science) 5
In the context of big data governance, what is 'data lineage'?