ACP Data Engineering & Workflow Automation Flashcards

Which pandas method reads a CSV file into a DataFrame and can handle large files by specifying a `chunksize` parameter?

Answer: pd.read_csv()

`pd.read_csv()` is the standard function for loading CSV data into a DataFrame, and its `chunksize` parameter returns an iterator of DataFrame chunks for memory-efficient processing.

In an ETL pipeline using pandas, which operation is used to combine two DataFrames based on a shared key column (similar to a SQL JOIN)?

Answer: pd.merge()

`pd.merge()` performs database-style joins between DataFrames on one or more key columns, supporting inner, outer, left, and right join types.

Which Python library provides the `Pipeline` class to chain preprocessing steps and a final estimator into a single reusable workflow object?

Answer: scikit-learn

scikit-learn's `Pipeline` chains transformers and a final estimator so that `fit` and `predict` calls automatically apply all steps in sequence.

When processing a large dataset in chunks to avoid memory overflow, which pandas parameter in `read_csv()` controls how many rows are loaded per iteration?

Answer: chunksize

Setting `chunksize=N` in `pd.read_csv()` returns a `TextFileReader` iterator where each iteration yields a DataFrame of N rows.

Which SQLAlchemy function is used alongside pandas `read_sql()` to connect to a database and execute SQL queries into a DataFrame?

Answer: create_engine()

`create_engine('dialect+driver://user:pass@host/db')` from SQLAlchemy creates a connection engine that pandas `read_sql()` uses to execute queries and return results as a DataFrame.

In pandas, which method writes a DataFrame to a SQL database table using a SQLAlchemy engine?

Answer: df.to_sql()

`df.to_sql('table_name', engine, if_exists='replace')` writes DataFrame contents to a SQL table, with options to append, replace, or fail if the table exists.