Databricks Certified Data Engineer Associate Auto Loader and Data Ingestion 1 — Questions and Answers
Question 1: What is the primary advantage of using Auto Loader over `spark.read` for cloud storage ingestion?
- It supports more file formats than spark.read
- It automatically identifies and processes only new files incrementally (Correct answer)
- It provides faster read speeds for all existing files
- It automatically compresses ingested data before writing
Correct answer: It automatically identifies and processes only new files incrementally
Auto Loader incrementally identifies and processes only new files as they arrive in cloud storage, avoiding reprocessing of previously seen files.
Question 2: Which Spark source format string is used to configure Auto Loader in a readStream call?
- autoloader
- cloudFiles (Correct answer)
- deltaStream
- incrementalFiles
Correct answer: cloudFiles
Auto Loader is accessed via `spark.readStream.format('cloudFiles')`, which is the Databricks-specific streaming source for cloud file ingestion.
Question 3: What is the purpose of the `cloudFiles.schemaLocation` option in Auto Loader?
- Specifies where to write output schema documentation
- Defines the location of manually provided input schema files
- Stores the inferred schema so it persists and evolves across stream restarts (Correct answer)
- Sets the schema registry endpoint for external validation
Correct answer: Stores the inferred schema so it persists and evolves across stream restarts
`cloudFiles.schemaLocation` tells Auto Loader where to persist the inferred schema on cloud storage, allowing it to reuse and evolve the schema across multiple stream restarts.
Question 4: What is the key difference between Auto Loader's 'directory listing' mode and 'file notification' mode?
- Directory listing is faster; file notification supports more file formats
- File notification uses cloud services (e.g., AWS SNS/SQS) for real-time updates; directory listing polls the directory periodically (Correct answer)
- Directory listing works with Delta tables only; file notification works with raw files
- File notification is the default mode and directory listing must be explicitly enabled
Correct answer: File notification uses cloud services (e.g., AWS SNS/SQS) for real-time updates; directory listing polls the directory periodically
File notification mode sets up cloud event infrastructure (like AWS SNS + SQS) so new file arrivals trigger processing immediately, while directory listing scans the directory on each trigger.
Question 5: When Auto Loader detects new columns in incoming files with schema evolution enabled, which option controls this behavior?
- cloudFiles.allowNewColumns
- cloudFiles.schemaEvolutionMode (Correct answer)
- cloudFiles.evolveSchema
- cloudFiles.inferColumnTypes
Correct answer: cloudFiles.schemaEvolutionMode
`cloudFiles.schemaEvolutionMode` controls how Auto Loader handles schema changes, with options such as 'addNewColumns', 'rescue', and 'failOnNewColumns'.
Question 6: Which of the following is a REQUIRED option when writing an Auto Loader stream output?
- checkpointLocation (Correct answer)
- cloudFiles.format
- cloudFiles.schemaLocation
- cloudFiles.maxFilesPerTrigger
Correct answer: checkpointLocation
`checkpointLocation` is required for all Structured Streaming writes, including Auto Loader, to enable fault tolerance and exactly-once processing guarantees.
Question 7: What happens to files already present in a directory when Auto Loader is started for the very first time?
- They are skipped; only files arriving after startup are processed
- They are processed along with all subsequent new files that arrive (Correct answer)
- Auto Loader fails with an error if existing files are detected
- They are automatically moved to an archive folder before processing begins
Correct answer: They are processed along with all subsequent new files that arrive
On its first run, Auto Loader processes all existing files in the source directory and then continues to pick up new files as they arrive.
Databricks Certified Data Engineer Associate Auto Loader and Data Ingestion 1
What is the primary advantage of using Auto Loader over `spark.read` for cloud storage ingestion?