Apache Spark Spark Performance Tuning 2

Question 1

What is the benefit of using Parquet over CSV in Spark?

Accepted Answer

Parquet is columnar, enabling predicate pushdown and efficient column pruning

Answer

Parquet's columnar storage enables Spark to read only needed columns and push filters to the storage layer, drastically reducing I/O.

Question 2

What is the Tungsten execution engine in Spark?

Accepted Answer

A low-level execution engine that uses off-heap memory and code generation for performance

Answer

Tungsten is Spark's physical execution engine that uses off-heap memory management and whole-stage code generation to maximize performance.

Question 3

Which Spark configuration sets the amount of memory allocated per executor?

Accepted Answer

spark.executor.memory

Answer

spark.executor.memory sets the amount of memory (e.g., '4g') allocated to each Spark executor JVM process.

Question 4

What is predicate pushdown in Spark SQL?

Accepted Answer

Moving filter conditions to earlier stages of the query plan to reduce data processed

Answer

Predicate pushdown moves filter conditions as close to the data source as possible, reducing the amount of data read from disk.

Question 5

What does the spark.memory.fraction configuration control?

Accepted Answer

The fraction of JVM heap used for Spark's execution and storage memory combined

Answer

spark.memory.fraction (default 0.6) defines the fraction of JVM heap available for Spark's unified execution and storage memory pool.

Apache Spark Practice Test