Apache Spark Spark Performance Tuning 2 — Questions and Answers
Question 1: What is the benefit of using Parquet over CSV in Spark?
- Parquet is human-readable, making debugging easier
- Parquet is columnar, enabling predicate pushdown and efficient column pruning (Correct answer)
- Parquet files are smaller because they use ZIP compression
- Parquet supports streaming reads while CSV does not
Correct answer: Parquet is columnar, enabling predicate pushdown and efficient column pruning
Parquet's columnar storage enables Spark to read only needed columns and push filters to the storage layer, drastically reducing I/O.
Question 2: What is the Tungsten execution engine in Spark?
- A new streaming engine replacing DStream
- A low-level execution engine that uses off-heap memory and code generation for performance (Correct answer)
- A GPU-accelerated computation backend
- A distributed SQL planner for Spark SQL
Correct answer: A low-level execution engine that uses off-heap memory and code generation for performance
Tungsten is Spark's physical execution engine that uses off-heap memory management and whole-stage code generation to maximize performance.
Question 3: Which Spark configuration sets the amount of memory allocated per executor?
- spark.executor.cores
- spark.executor.memory (Correct answer)
- spark.driver.memory
- spark.memory.fraction
Correct answer: spark.executor.memory
spark.executor.memory sets the amount of memory (e.g., '4g') allocated to each Spark executor JVM process.
Question 4: What is predicate pushdown in Spark SQL?
- Moving filter conditions to earlier stages of the query plan to reduce data processed (Correct answer)
- Pushing aggregation predicates to the driver for central computation
- Applying filters after all joins have been completed
- Sending filter conditions to the cluster manager for partition pruning
Correct answer: Moving filter conditions to earlier stages of the query plan to reduce data processed
Predicate pushdown moves filter conditions as close to the data source as possible, reducing the amount of data read from disk.
Question 5: What does the spark.memory.fraction configuration control?
- The fraction of executor memory reserved for user data caching only
- The fraction of JVM heap used for Spark's execution and storage memory combined (Correct answer)
- The ratio of driver memory to executor memory
- The percentage of disk space allocated for spill files
Correct answer: The fraction of JVM heap used for Spark's execution and storage memory combined
spark.memory.fraction (default 0.6) defines the fraction of JVM heap available for Spark's unified execution and storage memory pool.
Question 6: What is whole-stage code generation in Spark?
- Generating SQL code from DataFrame transformations
- Compiling an entire pipeline of operators into a single Java function to reduce interpretation overhead (Correct answer)
- Auto-generating Spark jobs from user configuration files
- Pre-compiling Spark applications before cluster submission
Correct answer: Compiling an entire pipeline of operators into a single Java function to reduce interpretation overhead
Whole-stage code generation compiles multiple query operators into a single optimized function, eliminating virtual function calls and improving CPU efficiency.
Apache Spark Spark Performance Tuning 2
What is the benefit of using Parquet over CSV in Spark?