Data Science with Python Data Cleaning and Preparation Flashcards

You are cleaning a dataset of customer information in a pandas DataFrame named 'df'. The 'phone_number' column contains numbers in various formats (e.g., '(123) 456-7890', '123.456.7890', '1234567890'). Which of the following is the most effective approach to standardize all phone numbers to the format '1234567890'?

Answer: Using the `.str.replace()` method with a regular expression to remove all non-digit characters.

The `.str.replace()` method in pandas is designed for vectorized string operations and, when combined with a regular expression like `r'\D'`, it can efficiently remove all non-digit characters from the entire column at once. This is more efficient and idiomatic than looping and more flexible than `astype()` which would fail on non-numeric characters. `fillna()` is for handling missing values, not for reformatting existing ones.

A data scientist is preparing a dataset for a machine learning model. They suspect the presence of outliers in a feature column that is approximately normally distributed. Which of the following methods is a common statistical approach to identify these outliers?

Answer: Calculating the Z-score for each data point and identifying values with an absolute Z-score greater than a threshold (commonly 3).

The Z-score measures how many standard deviations a data point is from the mean. For data that is normally distributed, a common rule of thumb is to consider data points with an absolute Z-score of 3 or more as outliers. This is a standard and effective method for outlier detection in such distributions.

While cleaning a pandas DataFrame, you discover that a column 'start_date', which should contain dates, is currently of the `object` dtype. Which pandas function is specifically designed to convert such a column to a proper datetime format?

Answer: pd.to_datetime(df['start_date'])

The `pd.to_datetime()` function is the most robust and recommended method for converting an array-like object of strings or other date representations into a pandas DatetimeIndex or Series. It can intelligently parse many different date formats. While `astype('datetime64[ns]')` can work, `to_datetime` is more flexible and powerful, especially with mixed formats or errors. `to_numeric` is for converting to numbers, not dates.

You have a DataFrame `sales_data` with duplicate rows based on an 'order_id' column. You need to remove these duplicates, keeping only the most recent entry for each 'order_id', which corresponds to the last occurrence in the DataFrame. Which line of code will achieve this?

Answer: sales_data.drop_duplicates(subset=['order_id'], keep='last')

The `drop_duplicates()` method is used to remove duplicate rows. The `subset` parameter allows you to specify which columns to consider for identifying duplicates. The `keep` parameter determines which duplicate to keep; `keep='last'` specifically retains the last occurrence of each duplicate row and removes the preceding ones.

A DataFrame contains a 'rating' column with some missing values represented as `np.nan`. The goal is to replace these missing values with the average rating of the column. Which of the following pandas operations correctly performs this imputation?

Answer: df['rating'].fillna(df['rating'].mean())

The `fillna()` method is the standard way to replace missing values (NA/NaN) in pandas. By passing `df['rating'].mean()` as the argument, it calculates the mean of the non-missing values in the 'rating' column and uses that result to fill all the `np.nan` entries.

Which of the following scenarios is the primary reason for converting a column's data type from `object` to a more specific type like `int` or `float` during data cleaning?

Answer: To enable mathematical operations and improve memory efficiency.

Columns with an `object` dtype often store strings, which prevents numerical calculations. Converting them to `int` or `float` using methods like `astype()` or `pd.to_numeric()` is essential for performing mathematical and statistical operations. Additionally, numeric types are generally more memory-efficient than object types.