Apache Spark Spark Performance Tuning

Question 1

What is data skew in Apache Spark and why is it a problem?

Accepted Answer

Uneven data distribution across partitions causing some tasks to take much longer than others

Answer

An imbalanced ratio of cores to memory across executors

Answer

Incorrect sorting of data that causes inaccurate results

Answer

A bug in the Spark scheduler that misallocates tasks

Question 2

What does the broadcast join hint do in Spark SQL?

Accepted Answer

Replicates a small DataFrame to all executors to avoid shuffling the large DataFrame

Answer

Distributes query results to all nodes

Answer

Forces Spark to use a merge-sort join algorithm

Answer

Broadcasts the query plan to all nodes for parallel planning

Question 3

What is the purpose of the spark.sql.shuffle.partitions configuration?

Accepted Answer

Controls the number of partitions used for shuffle operations like joins and aggregations

Answer

Sets the number of partitions when reading data from files

Answer

Defines the maximum partition size in bytes

Answer

Sets the number of concurrent shuffle readers

Question 4

What is speculative execution in Apache Spark?

Accepted Answer

Running duplicate copies of slow tasks on other nodes to handle stragglers

Answer

Pre-running tasks before data arrives for lower latency

Answer

Predicting job completion time before submission

Answer

Scheduling tasks optimistically without resource checks

Question 5

Which of the following strategies helps avoid data skew in a join operation?

Accepted Answer

Salting the join key with a random prefix

Answer

Increasing shuffle partitions

Answer

Using broadcast join for both sides

Answer

Reducing the number of executors

Question 6

What does Adaptive Query Execution (AQE) do in Spark 3.x?

Accepted Answer

Re-optimizes query plans at runtime based on actual statistics collected during execution

Answer

Automatically tunes executor memory based on workload

Answer

Automatically caches frequently accessed DataFrames

Answer

Distributes tasks to the least-loaded executors dynamically