Apache Spark Spark Performance Tuning 1 — Questions and Answers
Question 1: What is data skew in Apache Spark and why is it a problem?
- Uneven data distribution across partitions causing some tasks to take much longer than others (Correct answer)
- An imbalanced ratio of cores to memory across executors
- Incorrect sorting of data that causes inaccurate results
- A bug in the Spark scheduler that misallocates tasks
Correct answer: Uneven data distribution across partitions causing some tasks to take much longer than others
Data skew occurs when partitions have unequal sizes, making some tasks significantly slower and creating bottlenecks in the job.
Question 2: What does the broadcast join hint do in Spark SQL?
- Distributes query results to all nodes
- Replicates a small DataFrame to all executors to avoid shuffling the large DataFrame (Correct answer)
- Forces Spark to use a merge-sort join algorithm
- Broadcasts the query plan to all nodes for parallel planning
Correct answer: Replicates a small DataFrame to all executors to avoid shuffling the large DataFrame
Broadcast join sends a copy of the smaller DataFrame to all executor nodes, eliminating the expensive shuffle for the larger DataFrame.
Question 3: What is the purpose of the spark.sql.shuffle.partitions configuration?
- Sets the number of partitions when reading data from files
- Controls the number of partitions used for shuffle operations like joins and aggregations (Correct answer)
- Defines the maximum partition size in bytes
- Sets the number of concurrent shuffle readers
Correct answer: Controls the number of partitions used for shuffle operations like joins and aggregations
spark.sql.shuffle.partitions controls the number of partitions created after a shuffle (default 200), affecting performance of joins and aggregations.
Question 4: What is speculative execution in Apache Spark?
- Pre-running tasks before data arrives for lower latency
- Running duplicate copies of slow tasks on other nodes to handle stragglers (Correct answer)
- Predicting job completion time before submission
- Scheduling tasks optimistically without resource checks
Correct answer: Running duplicate copies of slow tasks on other nodes to handle stragglers
Speculative execution launches duplicate copies of straggler tasks on other nodes; whichever finishes first provides the result.
Question 5: Which of the following strategies helps avoid data skew in a join operation?
- Increasing shuffle partitions
- Salting the join key with a random prefix (Correct answer)
- Using broadcast join for both sides
- Reducing the number of executors
Correct answer: Salting the join key with a random prefix
Salting adds a random prefix to skewed keys, distributing hot keys across multiple partitions to avoid overloading a single task.
Question 6: What does Adaptive Query Execution (AQE) do in Spark 3.x?
- Automatically tunes executor memory based on workload
- Re-optimizes query plans at runtime based on actual statistics collected during execution (Correct answer)
- Automatically caches frequently accessed DataFrames
- Distributes tasks to the least-loaded executors dynamically
Correct answer: Re-optimizes query plans at runtime based on actual statistics collected during execution
AQE re-optimizes the query plan at runtime using actual partition statistics, enabling better join strategy selection and skew handling.
Apache Spark Spark Performance Tuning 1
What is data skew in Apache Spark and why is it a problem?