Data Engineering Data Serialization Formats 2 — Questions and Answers
Question 1: Which encoding technique does Parquet use for columns with few distinct values (low cardinality)?
- Run-length encoding
- Delta encoding
- Dictionary encoding (Correct answer)
- Bit-packing
Correct answer: Dictionary encoding
Dictionary encoding replaces repeated values with integer references to a small dictionary, which is highly effective for low-cardinality columns where the same values appear many times.
Question 2: What is the primary benefit of reading columnar formats like Parquet in analytical query engines?
- Faster row inserts and updates
- Only relevant columns are read from disk, reducing I/O significantly (Correct answer)
- Better support for ACID transactions
- Easier schema management across teams
Correct answer: Only relevant columns are read from disk, reducing I/O significantly
Columnar storage allows query engines to read only the columns they need and skip all others on disk, dramatically reducing I/O and speeding up analytical queries.
Question 3: Which serialization format is most widely used for data interchange with REST APIs?
- Avro
- Parquet
- ORC
- JSON (Correct answer)
Correct answer: JSON
JSON is the de facto standard for REST API data interchange due to its human-readable nature, wide language support, and native compatibility with web technologies.
Question 4: What is Protocol Buffers (protobuf) primarily designed for?
- Human-readable data storage and editing
- Efficient binary serialization for structured data across languages (Correct answer)
- Columnar analytical storage in data lakes
- Schema-free document storage in NoSQL databases
Correct answer: Efficient binary serialization for structured data across languages
Protocol Buffers is Google's binary serialization format designed for efficient, language-neutral structured data serialization with small payload size and fast parsing.
Question 5: Which Parquet feature allows query engines to skip entire row groups based on filter conditions without reading the data?
- Column pruning
- Row group filtering using min/max statistics (Correct answer)
- Predicate caching
- Schema projection
Correct answer: Row group filtering using min/max statistics
Parquet stores min/max statistics for each row group and column chunk, allowing query engines to skip entire row groups when filter conditions cannot match any row in that group.
Question 6: What is the key advantage of Avro for Apache Kafka message serialization compared to JSON?
- It is human-readable for easy debugging
- It supports columnar storage for faster reads
- Its compact binary format and schema evolution support via a schema registry (Correct answer)
- It automatically partitions Kafka topics
Correct answer: Its compact binary format and schema evolution support via a schema registry
Avro's compact binary encoding reduces Kafka message size for higher throughput, and its schema evolution support with a schema registry allows producers and consumers to evolve independently.
Question 7: What is the difference between backward compatibility and forward compatibility in schema evolution?
- They are the same concept applied in different systems
- Backward: new schema reads old data; Forward: old schema reads new data (Correct answer)
- Backward: old schema reads new data; Forward: new schema reads old data
- They refer to time-travel queries in data lakes
Correct answer: Backward: new schema reads old data; Forward: old schema reads new data
Backward compatibility means a newer schema version can read data written by an older schema, while forward compatibility means an older schema can read data written by a newer schema.
Data Engineering Data Serialization Formats 2
Which encoding technique does Parquet use for columns with few distinct values (low cardinality)?