CDP (Big Data Science) 4 — Questions and Answers
Question 1: In machine learning pipelines for big data, what is 'feature engineering' primarily concerned with?
- Selecting the optimal ML algorithm for a given dataset size
- Transforming raw data into informative representations that improve model performance (Correct answer)
- Distributing model training workloads across GPU clusters
- Monitoring model predictions in production for concept drift
Correct answer: Transforming raw data into informative representations that improve model performance
Feature engineering involves selecting, transforming, and creating input variables from raw data to improve the predictive power of machine learning models.
Question 2: What does 'data lake' refer to in modern big data architectures?
- A relational database optimized for storing structured transactional data at scale
- A centralized repository that stores raw data in its native format until needed (Correct answer)
- A message queue that buffers streaming data before it reaches the warehouse
- A distributed cache layer that accelerates queries on frequently accessed datasets
Correct answer: A centralized repository that stores raw data in its native format until needed
A data lake stores vast amounts of raw, unstructured, semi-structured, and structured data in its native format, deferring structure definition until query time.
Question 3: Which concept describes the practice of moving compute to where data resides rather than moving data to compute in big data systems?
- Data virtualization
- Data locality (Correct answer)
- Data federation
- Data sharding
Correct answer: Data locality
Data locality minimizes network I/O by scheduling computation on nodes that already hold the required data, a core principle in Hadoop's MapReduce scheduler.
Question 4: In stream processing, what is the difference between 'event time' and 'processing time'?
- Event time is when the system processes the event; processing time is when the event was generated
- Event time is when the event occurred at the source; processing time is when the system receives and processes it (Correct answer)
- Event time applies only to batch jobs; processing time applies to real-time streams
- Event time is measured in UTC; processing time uses local server time
Correct answer: Event time is when the event occurred at the source; processing time is when the system receives and processes it
Event time is the timestamp embedded in the event by the source system, while processing time is the wall-clock time when the stream processor receives the event.
Question 5: What is 'schema-on-read' as opposed to 'schema-on-write' in big data storage?
- Schema-on-read validates data structure only when the data is ingested into the system
- Schema-on-read applies structure to data at query time rather than enforcing it during ingestion (Correct answer)
- Schema-on-read requires all files to use self-describing formats like Avro
- Schema-on-read forces users to define column types before any data is loaded
Correct answer: Schema-on-read applies structure to data at query time rather than enforcing it during ingestion
Schema-on-read defers schema application to query time, allowing raw data to be stored flexibly and interpreted with different schemas as needed.
Question 6: Which technique in big data preprocessing addresses class imbalance in training datasets?
- Dimensionality reduction using PCA
- Oversampling minority classes or undersampling majority classes (e.g., SMOTE) (Correct answer)
- Normalizing feature values to a 0–1 range
- Removing duplicate records from the training set
Correct answer: Oversampling minority classes or undersampling majority classes (e.g., SMOTE)
Techniques like SMOTE (Synthetic Minority Oversampling Technique) generate synthetic samples for minority classes to balance the training distribution.
Question 7: What is 'partitioning' in distributed big data storage systems and why is it important?
- Encrypting data at rest by dividing it into separately keyed segments
- Dividing data across multiple nodes based on a key to enable parallel processing and scalability (Correct answer)
- Compressing data files into independently decodable chunks for storage efficiency
- Replicating datasets across geographically distributed data centers
Correct answer: Dividing data across multiple nodes based on a key to enable parallel processing and scalability
Partitioning distributes data across nodes based on a partition key, enabling parallelism, load balancing, and horizontal scalability in distributed systems.
CDP (Big Data Science) 4
In machine learning pipelines for big data, what is 'feature engineering' primarily concerned with?