CDS Automation & Scripting 3 — Questions and Answers
Question 1: A data steward needs to automate the detection of duplicate records in a large dataset. Which Python library is best suited for this at scale?
- csv module
- pandas (Correct answer)
- tkinter
- smtplib
Correct answer: pandas
pandas provides efficient DataFrame operations including drop_duplicates() and duplicated() methods for large-scale deduplication.
Question 2: When writing an automation script that processes sensitive PII, which practice is most critical for compliance?
- Log all PII values for debugging
- Mask or tokenize PII before logging or displaying (Correct answer)
- Store PII in plain-text log files
- Skip validation for PII fields to avoid exposure
Correct answer: Mask or tokenize PII before logging or displaying
Masking or tokenizing PII in logs and outputs prevents accidental exposure while still allowing debugging.
Question 3: Which approach best ensures that an automated data quality script fails safely when it cannot connect to the database?
- Use a bare except clause to catch all errors
- Use try/except blocks with specific exception types and send an alert (Correct answer)
- Print an error message and continue execution
- Ignore connection errors and process local cache
Correct answer: Use try/except blocks with specific exception types and send an alert
Specific exception handling combined with alerting ensures errors are caught precisely and the team is notified.
Question 4: A data steward is automating data lineage tracking. Which metadata should the script capture at minimum for each transformation step?
- Only the output dataset name
- Source, transformation logic, timestamp, and output dataset (Correct answer)
- Only the user who ran the script
- The server hostname only
Correct answer: Source, transformation logic, timestamp, and output dataset
Complete lineage requires capturing source, transformation, output, and timing to reconstruct the data's journey.
Question 5: In automated ETL pipelines, what is the main advantage of using parameterized SQL queries over string concatenation?
- They execute faster on all databases
- They prevent SQL injection attacks (Correct answer)
- They automatically optimize query plans
- They require less database permissions
Correct answer: They prevent SQL injection attacks
Parameterized queries separate SQL code from data inputs, preventing injection attacks entirely.
Question 6: A data steward wants to version-control automation scripts used by the team. Which tool and practice should be adopted?
- Store scripts in shared network folders
- Use Git with meaningful commit messages and branching strategy (Correct answer)
- Email script updates to team members
- Store only the latest version and overwrite old ones
Correct answer: Use Git with meaningful commit messages and branching strategy
Git provides full history, collaboration, rollback capability, and branching for team-based script development.
Question 7: When an automated data pipeline completes successfully, which notification mechanism is most appropriate for operational awareness?
- Write a log entry only
- Send a structured alert to a monitoring platform or messaging channel (e.g., Slack, PagerDuty) (Correct answer)
- Require a manual sign-off before next run
- Display output on the terminal only
Correct answer: Send a structured alert to a monitoring platform or messaging channel (e.g., Slack, PagerDuty)
Integrating with monitoring platforms ensures the operations team is proactively informed of pipeline status.
CDS Automation & Scripting 3
A data steward needs to automate the detection of duplicate records in a large dataset.
Which Python library is best suited for this at scale?