Databricks Certified Associate Developer for Apache Spark — Questions and Answers
Question 1: What is dynamic resource allocation in Spark?
- Distributing data across nodes based on access frequency
- Automatically adjusting executor memory based on task requirements
- Automatically partitioning data based on cluster size
- Scaling the number of executors up and down based on workload (Correct answer)
Correct answer: Scaling the number of executors up and down based on workload
Dynamic resource allocation allows Spark to add executors when tasks are queued and remove idle executors to free cluster resources.
Question 2: What is dynamic partition pruning in Spark 3.x?
- Filtering partitions of a fact table at runtime using values from a dimension table join (Correct answer)
- Automatically reducing partition count after shuffles
- Pruning null values from partitioned datasets
- Removing empty partitions from the output
Correct answer: Filtering partitions of a fact table at runtime using values from a dimension table join
Dynamic partition pruning pushes the result of a dimension table filter into the fact table scan, skipping irrelevant partitions at runtime.
Question 3: Which Spark SQL function removes duplicate rows from a DataFrame?
- deduplicate()
- removeDups()
- dropDuplicates() (Correct answer)
- unique()
Correct answer: dropDuplicates()
dropDuplicates() removes duplicate rows, optionally considering only a subset of columns.
Question 4: How does Spark achieve fault tolerance with RDDs?
- By recomputing lost partitions using lineage information (Correct answer)
- By writing all intermediate data to disk
- By maintaining checkpoints after every transformation
- By replicating data across multiple nodes
Correct answer: By recomputing lost partitions using lineage information
Spark achieves fault tolerance by recomputing lost RDD partitions using the recorded lineage of transformations.
Question 5: What is a DStream in Spark Streaming?
- A fault-tolerant message queue
- A distributed stream of DataFrames
- A real-time SQL query engine
- A sequence of RDDs representing a continuous stream of data (Correct answer)
Correct answer: A sequence of RDDs representing a continuous stream of data
A DStream (Discretized Stream) is a continuous sequence of RDDs, each representing data from a time interval.
Question 6: What is a Spark Task?
- A complete Spark job submitted by the driver
- A scheduled operation that runs on a timer
- A sub-DAG of transformations within a single stage
- A unit of work sent to one executor to process one data partition (Correct answer)
Correct answer: A unit of work sent to one executor to process one data partition
A Task is the smallest unit of work in Spark, processing one data partition and running on a single executor core.
Question 7: What is data skew in Apache Spark and why is it a problem?
- Uneven data distribution across partitions causing some tasks to take much longer than others (Correct answer)
- Incorrect sorting of data that causes inaccurate results
- A bug in the Spark scheduler that misallocates tasks
- An imbalanced ratio of cores to memory across executors
Correct answer: Uneven data distribution across partitions causing some tasks to take much longer than others
Data skew occurs when partitions have unequal sizes, making some tasks significantly slower and creating bottlenecks in the job.
Question 8: What is lineage in Apache Spark?
- The log of actions performed on a cluster
- The dependency tree of Spark executors
- The order in which jobs are executed
- The record of transformations used to build an RDD from base data (Correct answer)
Correct answer: The record of transformations used to build an RDD from base data
Lineage is the sequence of transformations that were applied to create an RDD, allowing Spark to recompute lost partitions.
Question 9: Spark Streaming's fundamental abstraction is
- Dstream (Correct answer)
- Shared Variable
- None of the above
- RDD
Correct answer: Dstream
DStream (Discretized Stream) is the fundamental abstraction in Spark Streaming, representing a continuous stream of data. Internally, a DStream is represented as a sequence of RDDs, where each RDD contains data from a small time interval. This abstraction allows Spark to process live data in micro-batches.
Question 10: How does Spark handle executor failures during a job?
- The driver takes over the computations from the failed executor
- Spark re-schedules the failed tasks on other available executors using RDD lineage (Correct answer)
- The job fails immediately when any executor is lost
- Spark pauses the job and waits for the executor to recover
Correct answer: Spark re-schedules the failed tasks on other available executors using RDD lineage
When an executor fails, Spark reschedules its tasks on other executors and recomputes any lost RDD partitions using lineage.
Question 11: Which GraphX edge partitioning strategy assigns edges to partitions based solely on the source vertex ID?
- EdgePartition1D (Correct answer)
- EdgePartition2D
- CanonicalRandomVertexCut
- RandomVertexCut
Correct answer: EdgePartition1D
EdgePartition1D assigns edges to partitions by hashing only the source vertex ID, colocating all outgoing edges from the same vertex in one partition.
Question 12: Which of these creates an RDD from an existing Scala/Python collection in Spark?
- sc.textFile()
- sc.parallelize() (Correct answer)
- sc.createRDD()
- sc.loadData()
Correct answer: sc.parallelize()
sc.parallelize() creates an RDD by distributing an existing local collection across the cluster.
Question 13: What is the purpose of the VectorAssembler in Spark MLlib?
- Reduces the dimensionality of features
- Scales feature values to a standard range
- Encodes categorical features as binary vectors
- Combines multiple feature columns into a single vector column (Correct answer)
Correct answer: Combines multiple feature columns into a single vector column
VectorAssembler merges multiple numeric columns into a single feature vector required by Spark ML algorithms.
Question 14: What does the `triplets` property of a GraphX Graph return?
- Three separate RDDs for source, edge, and destination
- An RDD of EdgeTriplet objects containing source vertex, edge, and destination vertex attributes (Correct answer)
- A tuple of (VertexRDD, EdgeRDD, PartitionRDD)
- An array of three graphs representing different structural views
Correct answer: An RDD of EdgeTriplet objects containing source vertex, edge, and destination vertex attributes
The `triplets` property returns an RDD[EdgeTriplet[VD, ED]], where each triplet combines the source vertex, the connecting edge, and the destination vertex with all their attributes.
Question 15: What parameters are used to define window operation?
- State size, window length
- None of the above
- Window length, sliding interval (Correct answer)
- State size, sliding interval
Correct answer: Window length, sliding interval
Window operations in Spark Streaming allow computations over a sliding window of data. These operations are defined by two key parameters: the 'window length,' which determines the duration of the data included in the window, and the 'sliding interval,' which specifies how often the window slides. These parameters control how data is aggregated and processed over time.
Question 16: What is the function of withColumn() in Spark DataFrames?
- Removes a column from the DataFrame
- Adds a new column or replaces an existing column with the given column expression (Correct answer)
- Selects only the specified column
- Renames a column
Correct answer: Adds a new column or replaces an existing column with the given column expression
withColumn() returns a new DataFrame by adding or replacing a column using the provided expression.
Question 17: Which function converts a DataFrame column to a different data type in Spark SQL?
- df.col('x').asType()
- df.col('x').convert()
- df.col('x').cast() (Correct answer)
- df.col('x').transform()
Correct answer: df.col('x').cast()
cast() is used to convert a column to a specified data type, e.g., col('age').cast('integer').
Question 18: What is YARN in the context of Apache Spark?
- A distributed file system for storing Spark data
- A cluster resource manager that allocates containers for Spark executors and the driver (Correct answer)
- A scheduler plugin for Spark's DAGScheduler
- A data serialization format used in Spark
Correct answer: A cluster resource manager that allocates containers for Spark executors and the driver
YARN (Yet Another Resource Negotiator) is a Hadoop resource manager that Spark uses to allocate containers for executors and the application master.
Question 19: Which method waits for a streaming query to finish in Spark?
- query.block()
- query.awaitTermination() (Correct answer)
- query.join()
- query.wait()
Correct answer: query.awaitTermination()
awaitTermination() blocks the driver program until the streaming query is stopped or encounters an error.
Question 20: What is GBTClassifier in Spark MLlib?
- A Gaussian Bayesian Tree Classifier
- A generalized batch training algorithm
- A graph-based binary tree classifier
- A Gradient Boosted Trees Classifier for binary classification (Correct answer)
Correct answer: A Gradient Boosted Trees Classifier for binary classification
GBTClassifier implements gradient boosted trees for binary classification, which iteratively trains decision trees to correct previous errors.
Question 21: Which format is recommended for reading structured data in Spark SQL when schema inference is needed?
- Parquet
- CSV
- ORC
- JSON (Correct answer)
Correct answer: JSON
JSON allows Spark to automatically infer the schema during read, though Parquet is preferred for performance in production.
Question 22: Which method is used to persist an RDD in Spark?
- store()
- save()
- cache() or persist() (Correct answer)
- memorize()
Correct answer: cache() or persist()
cache() and persist() are used to persist an RDD; cache() is a shorthand for persist(StorageLevel.MEMORY_ONLY).
Question 23: What does the explode() function do in Spark SQL?
- Splits a string column into multiple columns
- Increases the number of partitions in the DataFrame
- Raises an exception if data is malformed
- Converts a column of arrays or maps into multiple rows (Correct answer)
Correct answer: Converts a column of arrays or maps into multiple rows
explode() takes a column containing an array or map and creates a new row for each element.
Question 24: What does spark-submit --master yarn --deploy-mode cluster do?
- Runs a Spark job on the local machine using all available cores
- Deploys a Spark application to every node in the YARN cluster simultaneously
- Submits a Spark application to a standalone Spark cluster in cluster mode
- Submits a Spark application to a YARN cluster with the driver running on a YARN container (Correct answer)
Correct answer: Submits a Spark application to a YARN cluster with the driver running on a YARN container
This command submits a Spark application to YARN and runs the Driver in a YARN ApplicationMaster container on the cluster.
Question 25: How do you write a Spark DataFrame to a Parquet file?
- df.write.parquet('path') (Correct answer)
- df.export('path', format='parquet')
- df.save('path', 'parquet')
- df.to_parquet('path')
Correct answer: df.write.parquet('path')
df.write.parquet('path') saves the DataFrame to the specified path in Parquet format.
Question 26: What is a Spark Stage?
- A group of tasks that can be executed in parallel without a shuffle
- A phase of the DAG separated by wide transformations (shuffle boundaries) (Correct answer)
- A single task running on one partition
- A container for multiple Spark jobs
Correct answer: A phase of the DAG separated by wide transformations (shuffle boundaries)
A Stage is a set of tasks that can be computed without shuffling data; stage boundaries are determined by wide transformations.
Question 27: In which Spark release was the dataset introduced?
- Spark 1.1
- Spark 1.6 (Correct answer)
- Spark 1.4.0
- Spark 2.1.0
Correct answer: Spark 1.6
The Dataset API was introduced in Apache Spark 1.6. It aimed to provide the best of both RDDs (strong typing and compile-time safety) and DataFrames (optimized execution and query planning) by offering a type-safe, object-oriented programming interface. This addition significantly improved developer productivity and performance.
Question 28: What is a sliding window in Spark Structured Streaming?
- A window that processes exactly one trigger interval at a time
- A window based on the number of records rather than time
- A window that moves forward by a fixed step smaller than the window duration, causing overlaps (Correct answer)
- A fixed-size window that never overlaps with adjacent windows
Correct answer: A window that moves forward by a fixed step smaller than the window duration, causing overlaps
A sliding window has a duration and a slide interval; if the slide is smaller than the window duration, windows overlap.
Question 29: What is the role of the SparkContext in a Spark application?
- Manages SQL queries
- Provides the DataFrame API
- Serves as the entry point for Spark functionality and represents the connection to a Spark cluster (Correct answer)
- Schedules streaming jobs
Correct answer: Serves as the entry point for Spark functionality and represents the connection to a Spark cluster
SparkContext is the main entry point for Spark Core API, representing the connection to a Spark cluster and used to create RDDs.
Question 30: What is Catalyst in Apache Spark?
- The query optimizer for Spark SQL (Correct answer)
- A machine learning library
- A streaming engine
- A storage manager for DataFrames
Correct answer: The query optimizer for Spark SQL
Catalyst is Spark SQL's extensible query optimizer that transforms logical plans into optimized physical plans.
Question 31: Which method forces Spark to recompute an RDD and remove its cached data?
- rdd.release()
- rdd.evict()
- rdd.unpersist() (Correct answer)
- rdd.clear()
Correct answer: rdd.unpersist()
unpersist() removes the RDD from the cache, freeing the memory for other computations.
Question 32: What are Spark Executors?
- Memory managers that handle executor JVM garbage collection
- JVM processes launched by the cluster manager that execute tasks and store RDD data (Correct answer)
- Processes on the driver node that compile the DAG into tasks
- Worker nodes in the cluster that manage data replication
Correct answer: JVM processes launched by the cluster manager that execute tasks and store RDD data
Executors are JVM processes running on worker nodes that execute tasks assigned by the Driver and cache RDD partitions in memory.
Question 33: What is the entry point for Spark SQL functionality in Spark 2.x and later?
- SQLContext
- SparkContext
- HiveContext
- SparkSession (Correct answer)
Correct answer: SparkSession
SparkSession is the unified entry point for Spark SQL, DataFrame, and Dataset APIs introduced in Spark 2.0.
Question 34: What happens when Spark runs out of memory for a shuffle and cannot fit data in memory?
- The job fails immediately with an OOM error
- Spark automatically increases executor memory
- Spark spills the excess data to disk (Correct answer)
- Spark drops the data and continues
Correct answer: Spark spills the excess data to disk
When execution memory is exhausted during a shuffle, Spark spills data to disk, which is slower but allows the job to continue.
Question 35: Which output mode in Structured Streaming writes only new rows added to the result table since the last trigger?
- Append (Correct answer)
- Update
- Complete
- Incremental
Correct answer: Append
Append mode writes only new rows appended to the result table since the last trigger to the sink.
Question 36: What is the Update output mode in Spark Structured Streaming?
- Writes the entire result table every trigger
- Writes only rows that were updated or newly added since the last trigger (Correct answer)
- Updates the previous output file in place
- Writes only rows that are new appends
Correct answer: Writes only rows that were updated or newly added since the last trigger
Update mode writes only the rows in the result table that were changed (inserted or updated) since the last trigger.
Question 37: What is the purpose of spark.default.parallelism?
- Sets the thread count within each executor JVM
- Controls the maximum number of concurrent tasks across all executors
- Sets the default number of RDD partitions for transformations that create new partitions (Correct answer)
- Defines the number of partitions for SQL shuffle operations
Correct answer: Sets the default number of RDD partitions for transformations that create new partitions
spark.default.parallelism sets the default number of partitions for RDD operations like reduceByKey and join when no explicit partition count is given.
Question 38: Which of the following is a valid trigger for Structured Streaming in Spark?
- ProcessingTime trigger (Correct answer)
- BatchSize trigger
- RowCount trigger
- EventTime trigger
Correct answer: ProcessingTime trigger
ProcessingTime trigger runs micro-batches at a specified interval, e.g., Trigger.ProcessingTime('10 seconds').
Question 39: In GraphX, what are the two fundamental components of a Property Graph?
- Edges and Paths
- Nodes and Relationships
- Vertices and Edges (Correct answer)
- Rows and Columns
Correct answer: Vertices and Edges
A Property Graph in GraphX consists of vertices (nodes) and edges (connections), each carrying associated user-defined properties.
Question 40: Which Spark MLlib class is used to convert categorical string labels to numeric indices?
- StringIndexer (Correct answer)
- OneHotEncoder
- CategoryMapper
- LabelEncoder
Correct answer: StringIndexer
StringIndexer encodes a string column of labels to a column of label indices, ordered by label frequencies.
Question 41: In which deploy mode does the Spark driver run on the machine where spark-submit is executed?
- Cluster mode
- Client mode (Correct answer)
- Standalone mode
- Local mode
Correct answer: Client mode
In client mode, the Driver runs on the machine that submitted the application, which is useful for interactive development but not recommended for production.
Question 42: What is a DataFrame in Apache Spark?
- An immutable list of serialized Java objects
- A streaming data source
- A distributed key-value store
- An RDD of Row objects with a named column schema (Correct answer)
Correct answer: An RDD of Row objects with a named column schema
A DataFrame is a distributed collection of data organized into named columns, conceptually equivalent to a database table.
Question 43: Which of the following statements regarding DataFrame is correct?
- None of the above
- DataFrames provide a more user-friendly API than RDDs. (Correct answer)
- Both the above
- DataFrame API have provision for compile-time type safety
Correct answer: DataFrames provide a more user-friendly API than RDDs.
DataFrames offer a higher-level, more structured, and user-friendly API compared to RDDs, resembling tables in a relational database. They provide schema information and allow for optimized execution plans, making data manipulation and querying more intuitive and efficient for many use cases. This improved usability is a key advantage over raw RDDs.
Question 44: Which of the following statements about Spark R is correct?
- It allows data scientists to analyze large datasets and interactively run jobs (Correct answer)
- It is the kernel of Spark
- It enables users to run SQL / HQL queries on the top of Spark.
- It is the scalable machine learning library which delivers efficiencies
Correct answer: It allows data scientists to analyze large datasets and interactively run jobs
SparkR is an R package that provides a lightweight frontend to use Apache Spark from R. It enables R users to interact with Spark, perform data analysis on large datasets, and leverage Spark's distributed computing capabilities within their familiar R environment. This integration empowers data scientists to scale their R workflows.
Question 45: What does the spark.memory.fraction configuration control?
- The ratio of driver memory to executor memory
- The fraction of executor memory reserved for user data caching only
- The percentage of disk space allocated for spill files
- The fraction of JVM heap used for Spark's execution and storage memory combined (Correct answer)
Correct answer: The fraction of JVM heap used for Spark's execution and storage memory combined
spark.memory.fraction (default 0.6) defines the fraction of JVM heap available for Spark's unified execution and storage memory pool.
Databricks Certified Associate Developer for Apache Spark
The Databricks Certified Associate Developer for Apache Spark certification validates proficiency in using Apache Spark and the Databricks Lakehouse Platform to complete introductory-level data engineering tasks, covering Spark architecture, SQL, DataFrames, performance tuning, and advanced processing features.
Exam Rules
- You can skip questions and return to them later
- Flag questions for review before submitting
- No feedback shown until you submit the entire exam
- Unanswered questions count as wrong — answer everything
- 10 pretest questions are mixed in and don't affect your score
- Timer auto-submits when time runs out
- Your progress is auto-saved every 30 seconds
Practice questions written with AI assistance and published by Nathan Beja, founder of Practice Test Geeks. They are not real exam questions. Exam facts are checked against the official sources linked below. Editorial policy
Sources & references
Exam facts on this page (fees, format, dates, passing scores) are checked against these official sources; the date is our last check. Editorial policy
- databricks.com — Test delivery, fees, exam format, eligibility, renewal, time limit, passing score, retake policy, score release, score validity · verified Nov 2026