Databricks Certified Data Engineer Associate Auto Loader and Data Ingestion 2 — Questions and Answers
Question 1: Which Auto Loader option limits processing to a maximum number of files per trigger interval?
- cloudFiles.maxBytesPerTrigger
- cloudFiles.maxFilesPerTrigger (Correct answer)
- trigger.maxFiles
- cloudFiles.batchSize
Correct answer: cloudFiles.maxFilesPerTrigger
`cloudFiles.maxFilesPerTrigger` caps the number of files Auto Loader will process in a single micro-batch, useful for controlling throughput and cluster load.
Question 2: Auto Loader's file notification mode on AWS relies on which combination of managed services?
- S3 + CloudWatch + Lambda
- S3 + SNS + SQS (Correct answer)
- S3 + Kinesis + DynamoDB
- S3 + EventBridge + Step Functions
Correct answer: S3 + SNS + SQS
On AWS, Auto Loader's file notification mode uses S3 event notifications published to an SNS topic, which fans out to an SQS queue that Auto Loader polls for new file events.
Question 3: When `cloudFiles.inferColumnTypes` is set to false in Auto Loader schema inference, how are columns typed?
- Auto Loader fails to start without an explicit schema
- All inferred columns default to StringType (Correct answer)
- Only primitive types are inferred; complex types are dropped
- Auto Loader uses the last known schema from schemaLocation
Correct answer: All inferred columns default to StringType
With `cloudFiles.inferColumnTypes=false`, Auto Loader infers the schema but types every column as StringType, which is the safest fallback for heterogeneous data.
Question 4: Which statement BEST compares COPY INTO versus Auto Loader for data ingestion?
- COPY INTO supports exactly-once semantics natively; Auto Loader does not
- Auto Loader is better suited for large-scale continuous streaming; COPY INTO is designed for idempotent batch loads (Correct answer)
- COPY INTO supports schema evolution; Auto Loader does not
- Auto Loader requires a Delta table target; COPY INTO works with any file format
Correct answer: Auto Loader is better suited for large-scale continuous streaming; COPY INTO is designed for idempotent batch loads
Auto Loader excels at large-scale, continuously arriving data with Structured Streaming, while COPY INTO is an idempotent SQL command suited for scheduled batch ingestion into Delta tables.
Question 5: Which of the following file formats is NOT natively supported as input by Auto Loader?
- JSON
- CSV
- Microsoft Excel (.xlsx) (Correct answer)
- Parquet
Correct answer: Microsoft Excel (.xlsx)
Auto Loader natively supports JSON, CSV, Parquet, Avro, ORC, text, and binary files, but does not support proprietary formats like Microsoft Excel (.xlsx).
Question 6: What is the purpose of the `rescuedDataColumn` option in Auto Loader?
- It stores rows or fields that could not be parsed due to schema mismatches in a dedicated column (Correct answer)
- It defines a backup storage location for files that failed to load
- It specifies a partition column used to segregate problematic data
- It enables automatic recovery and reprocessing of corrupted source files
Correct answer: It stores rows or fields that could not be parsed due to schema mismatches in a dedicated column
`rescuedDataColumn` causes Auto Loader to place any data that doesn't match the schema (unexpected columns, type mismatches) into a named column as a JSON string rather than dropping or failing.
Question 7: How does Auto Loader prevent duplicate file processing when a streaming job restarts after a failure?
- It relies on Delta Lake's ACID properties to detect and remove duplicates on write
- It uses a checkpoint directory to track previously processed files, ensuring each file is processed exactly once (Correct answer)
- Deduplication must be handled entirely by application-level logic after ingestion
- Auto Loader uses cloud storage object versioning to detect already-processed files
Correct answer: It uses a checkpoint directory to track previously processed files, ensuring each file is processed exactly once
Auto Loader's checkpoint directory stores offsets and the complete list of processed files, so on restart it knows exactly which files have already been ingested and skips them.
Databricks Certified Data Engineer Associate Auto Loader and Data Ingestion 2
Which Auto Loader option limits processing to a maximum number of files per trigger interval?