Data Science with Python Data Cleaning and Preparation Questions and Answers — Questions and Answers
Question 1: You are cleaning a dataset of customer information in a pandas DataFrame named 'df'. The 'phone_number' column contains numbers in various formats (e.g., '(123) 456-7890', '123.456.7890', '1234567890'). Which of the following is the most effective approach to standardize all phone numbers to the format '1234567890'?
- Using a for loop to iterate over each phone number and manually replace characters.
- Applying the `astype(int)` method to the column.
- Using the `.str.replace()` method with a regular expression to remove all non-digit characters. (Correct answer)
- Using the `fillna()` method to replace malformed numbers with a standard one.
Correct answer: Using the `.str.replace()` method with a regular expression to remove all non-digit characters.
The `.str.replace()` method in pandas is designed for vectorized string operations and, when combined with a regular expression like `r'\D'`, it can efficiently remove all non-digit characters from the entire column at once. This is more efficient and idiomatic than looping and more flexible than `astype()` which would fail on non-numeric characters. `fillna()` is for handling missing values, not for reformatting existing ones.
Question 2: A data scientist is preparing a dataset for a machine learning model. They suspect the presence of outliers in a feature column that is approximately normally distributed. Which of the following methods is a common statistical approach to identify these outliers?
- Calculating the median absolute deviation (MAD) and identifying values beyond a certain threshold.
- Calculating the Z-score for each data point and identifying values with an absolute Z-score greater than a threshold (commonly 3). (Correct answer)
- Grouping the data by value counts and identifying the least frequent values.
- Forward-filling the data to smooth out extreme values.
Correct answer: Calculating the Z-score for each data point and identifying values with an absolute Z-score greater than a threshold (commonly 3).
The Z-score measures how many standard deviations a data point is from the mean. For data that is normally distributed, a common rule of thumb is to consider data points with an absolute Z-score of 3 or more as outliers. This is a standard and effective method for outlier detection in such distributions.
Question 3: While cleaning a pandas DataFrame, you discover that a column 'start_date', which should contain dates, is currently of the `object` dtype. Which pandas function is specifically designed to convert such a column to a proper datetime format?
- df['start_date'].astype('datetime64[ns]')
- pd.to_datetime(df['start_date']) (Correct answer)
- df['start_date'].apply(lambda x: datetime.strptime(x, '%Y-%m-%d'))
- pd.to_numeric(df['start_date'])
Correct answer: pd.to_datetime(df['start_date'])
The `pd.to_datetime()` function is the most robust and recommended method for converting an array-like object of strings or other date representations into a pandas DatetimeIndex or Series. It can intelligently parse many different date formats. While `astype('datetime64[ns]')` can work, `to_datetime` is more flexible and powerful, especially with mixed formats or errors. `to_numeric` is for converting to numbers, not dates.
Question 4: You have a DataFrame `sales_data` with duplicate rows based on an 'order_id' column. You need to remove these duplicates, keeping only the most recent entry for each 'order_id', which corresponds to the last occurrence in the DataFrame. Which line of code will achieve this?
- sales_data.drop_duplicates(subset=['order_id'], keep='first')
- sales_data.drop_duplicates(subset=['order_id'], keep=False)
- sales_data[~sales_data.duplicated(subset=['order_id'], keep='last')]
- sales_data.drop_duplicates(subset=['order_id'], keep='last') (Correct answer)
Correct answer: sales_data.drop_duplicates(subset=['order_id'], keep='last')
The `drop_duplicates()` method is used to remove duplicate rows. The `subset` parameter allows you to specify which columns to consider for identifying duplicates. The `keep` parameter determines which duplicate to keep; `keep='last'` specifically retains the last occurrence of each duplicate row and removes the preceding ones.
Question 5: A DataFrame contains a 'rating' column with some missing values represented as `np.nan`. The goal is to replace these missing values with the average rating of the column. Which of the following pandas operations correctly performs this imputation?
- df['rating'].dropna()
- df['rating'].fillna(df['rating'].mean()) (Correct answer)
- df['rating'].replace(np.nan, 'mean')
- df['rating'][df['rating'].isnull()] = df['rating'].median()
Correct answer: df['rating'].fillna(df['rating'].mean())
The `fillna()` method is the standard way to replace missing values (NA/NaN) in pandas. By passing `df['rating'].mean()` as the argument, it calculates the mean of the non-missing values in the 'rating' column and uses that result to fill all the `np.nan` entries.
Question 6: Which of the following scenarios is the primary reason for converting a column's data type from `object` to a more specific type like `int` or `float` during data cleaning?
- To enable mathematical operations and improve memory efficiency. (Correct answer)
- To make the DataFrame display more aesthetically pleasing.
- To ensure all columns have the same number of unique values.
- To prepare the data for export to a JSON file, which requires numeric types.
Correct answer: To enable mathematical operations and improve memory efficiency.
Columns with an `object` dtype often store strings, which prevents numerical calculations. Converting them to `int` or `float` using methods like `astype()` or `pd.to_numeric()` is essential for performing mathematical and statistical operations. Additionally, numeric types are generally more memory-efficient than object types.
Data Science with Python Data Cleaning and Preparation Questions and Answers
You are cleaning a dataset of customer information in a pandas DataFrame named 'df'.
The 'phone_number' column contains numbers in various formats (e.g., '(123) 456-7890', '123.456.7890', '1234567890').
Which of the following is the most effective approach to standardize all phone numbers to the format '1234567890'?