Data Engineering Data Serialization Formats 1 — Questions and Answers
Question 1: What is the primary advantage of Apache Parquet over CSV for analytical workloads?
- It supports real-time data streaming
- It stores data in columnar format enabling faster analytical queries (Correct answer)
- It is human-readable and easier to debug
- It has no schema requirements
Correct answer: It stores data in columnar format enabling faster analytical queries
Parquet stores data in a columnar format, allowing analytical queries to read only the relevant columns and dramatically reducing I/O.
Question 2: Which serialization format uses binary encoding and requires a schema to serialize and deserialize data?
- JSON
- CSV
- Apache Avro (Correct answer)
- XML
Correct answer: Apache Avro
Apache Avro uses binary encoding and requires a schema (defined in JSON) to serialize and deserialize data, making it compact and efficient.
Question 3: What type of internal storage organization does ORC (Optimized Row Columnar) use?
- Pure row-based storage
- Pure columnar storage
- A hybrid of row stripes containing columnar data (Correct answer)
- Key-value based storage
Correct answer: A hybrid of row stripes containing columnar data
ORC splits data into stripes (row groups) and within each stripe stores data in columnar format, combining the benefits of both row and columnar approaches.
Question 4: Which serialization format is most appropriate when humans need to inspect and edit data directly without special tools?
- Parquet
- Avro
- JSON (Correct answer)
- Protocol Buffers
Correct answer: JSON
JSON is a human-readable text format that can be opened and understood in any text editor, making it ideal for debugging and manual inspection.
Question 5: What does 'schema evolution' mean in the context of data serialization?
- Replacing an old schema with a completely new incompatible one
- The ability to modify a schema over time while maintaining compatibility with existing data (Correct answer)
- Automatically generating schemas from raw data
- Converting schemas between different file formats
Correct answer: The ability to modify a schema over time while maintaining compatibility with existing data
Schema evolution is the ability to modify a schema (e.g., add or remove fields) over time while still reading data written with older or newer schema versions.
Question 6: Which format provides the best compression ratios for analytical data due to columnar storage and value encoding techniques?
- CSV
- JSON
- XML
- Parquet (Correct answer)
Correct answer: Parquet
Parquet achieves excellent compression by storing similar data types together in columns, enabling encodings like dictionary encoding and run-length encoding that exploit data locality.
Question 7: In Apache Avro, where is the writer's schema typically stored to make files self-describing?
- In a separate .xsd file alongside the data
- In the file header of each Avro data file (Correct answer)
- In a YAML configuration file on the server
- In the application code only at runtime
Correct answer: In the file header of each Avro data file
Avro embeds the writer's schema in the header of each data file, ensuring the schema travels with the data and enabling self-describing files.
Data Engineering Data Serialization Formats 1
What is the primary advantage of Apache Parquet over CSV for analytical workloads?