ACP ACP Data Engineering & Workflow Automation 1 — Questions and Answers
Question 1: Which pandas method reads a CSV file into a DataFrame and can handle large files by specifying a `chunksize` parameter?
- pd.read_table()
- pd.read_csv() (Correct answer)
- pd.from_csv()
- pd.load_csv()
Correct answer: pd.read_csv()
`pd.read_csv()` is the standard function for loading CSV data into a DataFrame, and its `chunksize` parameter returns an iterator of DataFrame chunks for memory-efficient processing.
Question 2: In an ETL pipeline using pandas, which operation is used to combine two DataFrames based on a shared key column (similar to a SQL JOIN)?
- pd.concat()
- pd.merge() (Correct answer)
- pd.join_tables()
- pd.combine()
Correct answer: pd.merge()
`pd.merge()` performs database-style joins between DataFrames on one or more key columns, supporting inner, outer, left, and right join types.
Question 3: Which Python library provides the `Pipeline` class to chain preprocessing steps and a final estimator into a single reusable workflow object?
- pandas
- numpy
- scikit-learn (Correct answer)
- scipy
Correct answer: scikit-learn
scikit-learn's `Pipeline` chains transformers and a final estimator so that `fit` and `predict` calls automatically apply all steps in sequence.
Question 4: When processing a large dataset in chunks to avoid memory overflow, which pandas parameter in `read_csv()` controls how many rows are loaded per iteration?
- batch_size
- chunksize (Correct answer)
- nrows
- max_rows
Correct answer: chunksize
Setting `chunksize=N` in `pd.read_csv()` returns a `TextFileReader` iterator where each iteration yields a DataFrame of N rows.
Question 5: Which SQLAlchemy function is used alongside pandas `read_sql()` to connect to a database and execute SQL queries into a DataFrame?
- create_engine() (Correct answer)
- connect_db()
- open_session()
- make_connection()
Correct answer: create_engine()
`create_engine('dialect+driver://user:pass@host/db')` from SQLAlchemy creates a connection engine that pandas `read_sql()` uses to execute queries and return results as a DataFrame.
Question 6: In pandas, which method writes a DataFrame to a SQL database table using a SQLAlchemy engine?
- df.to_database()
- df.to_sql() (Correct answer)
- df.write_sql()
- df.export_sql()
Correct answer: df.to_sql()
`df.to_sql('table_name', engine, if_exists='replace')` writes DataFrame contents to a SQL table, with options to append, replace, or fail if the table exists.
ACP ACP Data Engineering & Workflow Automation 1
Which pandas method reads a CSV file into a DataFrame and can handle large files by specifying a `chunksize` parameter?