LFC Photon Engine Optimization 2 — Questions and Answers
Question 1: Which data format characteristic most benefits Photon's vectorized execution?
- Row-based storage with wide tables
- Columnar storage with contiguous memory layout (Correct answer)
- JSON semi-structured format
- CSV with mixed data types
Correct answer: Columnar storage with contiguous memory layout
Photon's vectorized execution processes batches of column values stored contiguously in memory, maximizing CPU cache efficiency.
Question 2: What happens to Photon execution when a query encounters a UDF (User Defined Function)?
- Photon skips the UDF and returns null values
- Photon falls back to the JVM-based Spark engine for UDF evaluation (Correct answer)
- Photon compiles the UDF into native code automatically
- The query fails with an unsupported operation error
Correct answer: Photon falls back to the JVM-based Spark engine for UDF evaluation
Photon cannot execute JVM-based UDFs natively and falls back to the standard Spark engine for those operations.
Question 3: In Photon, what is the primary purpose of predicate pushdown when reading Delta tables?
- To rewrite predicates as window functions
- To skip reading data files that don't match filter conditions (Correct answer)
- To push filters to the network layer
- To cache predicate results in Redis
Correct answer: To skip reading data files that don't match filter conditions
Predicate pushdown allows Photon to use Delta Lake's file statistics to skip entire data files that cannot contain matching rows.
Question 4: Which cluster configuration setting directly controls whether Photon is enabled on a Databricks cluster?
- spark.sql.vectorized.reader.enabled
- spark.databricks.photon.enabled (Correct answer)
- spark.executor.photon.mode
- databricks.runtime.photon=true
Correct answer: spark.databricks.photon.enabled
The spark.databricks.photon.enabled configuration property enables or disables the Photon native execution engine on a cluster.
Question 5: How does Photon handle NULL values differently from traditional Spark execution?
- Photon treats NULLs as zeros for arithmetic operations
- Photon uses validity bitmaps to track NULLs separately from data vectors (Correct answer)
- Photon skips rows containing NULLs during aggregations
- Photon converts NULLs to empty strings before processing
Correct answer: Photon uses validity bitmaps to track NULLs separately from data vectors
Photon maintains a separate validity bitmap alongside each data vector, allowing NULL-aware vectorized operations without branching on every value.
Question 6: Which scenario would most likely cause Photon to show the greatest performance improvement over standard Spark?
- A simple key-value lookup on a 100-row table
- A large aggregation with GROUP BY on billions of rows in Parquet files (Correct answer)
- A Python UDF that calls an external REST API
- A single-node job with no shuffling
Correct answer: A large aggregation with GROUP BY on billions of rows in Parquet files
Photon's vectorized native execution provides the largest gains on CPU-bound operations like aggregations over large datasets stored in columnar formats.
Question 7: What does the Photon query plan indicator 'photon=true' in the Spark UI signify?
- The entire query ran in Photon without fallback
- That specific operator was executed by the Photon engine (Correct answer)
- Photon cached the results for future queries
- The query used Photon's approximate aggregation mode
Correct answer: That specific operator was executed by the Photon engine
Individual operators in the query plan are annotated with photon=true when Photon executed that specific operation, while others may still run on JVM Spark.
LFC Photon Engine Optimization 2
Which data format characteristic most benefits Photon's vectorized execution?