Data Science with Python Data Cleaning and Preparation Questions and Answers

Question 1

You are cleaning a dataset of customer information in a pandas DataFrame named 'df'. The 'phone_number' column contains numbers in various formats (e.g., '(123) 456-7890', '123.456.7890', '1234567890'). Which of the following is the most effective approach to standardize all phone numbers to the format '1234567890'?

Accepted Answer

Using the `.str.replace()` method with a regular expression to remove all non-digit characters.

Answer

The `.str.replace()` method in pandas is designed for vectorized string operations and, when combined with a regular expression like `r'\D'`, it can efficiently remove all non-digit characters from the entire column at once. This is more efficient and idiomatic than looping and more flexible than `astype()` which would fail on non-numeric characters. `fillna()` is for handling missing values, not for reformatting existing ones.

Question 2

A data scientist is preparing a dataset for a machine learning model. They suspect the presence of outliers in a feature column that is approximately normally distributed. Which of the following methods is a common statistical approach to identify these outliers?

Accepted Answer

Calculating the Z-score for each data point and identifying values with an absolute Z-score greater than a threshold (commonly 3).

Answer

The Z-score measures how many standard deviations a data point is from the mean. For data that is normally distributed, a common rule of thumb is to consider data points with an absolute Z-score of 3 or more as outliers. This is a standard and effective method for outlier detection in such distributions.

Question 3

While cleaning a pandas DataFrame, you discover that a column 'start_date', which should contain dates, is currently of the `object` dtype. Which pandas function is specifically designed to convert such a column to a proper datetime format?

Accepted Answer

pd.to_datetime(df['start_date'])

Answer

The `pd.to_datetime()` function is the most robust and recommended method for converting an array-like object of strings or other date representations into a pandas DatetimeIndex or Series. It can intelligently parse many different date formats. While `astype('datetime64[ns]')` can work, `to_datetime` is more flexible and powerful, especially with mixed formats or errors. `to_numeric` is for converting to numbers, not dates.

Question 4

You have a DataFrame `sales_data` with duplicate rows based on an 'order_id' column. You need to remove these duplicates, keeping only the most recent entry for each 'order_id', which corresponds to the last occurrence in the DataFrame. Which line of code will achieve this?

Accepted Answer

sales_data.drop_duplicates(subset=['order_id'], keep='last')

Answer

The `drop_duplicates()` method is used to remove duplicate rows. The `subset` parameter allows you to specify which columns to consider for identifying duplicates. The `keep` parameter determines which duplicate to keep; `keep='last'` specifically retains the last occurrence of each duplicate row and removes the preceding ones.

Question 5

A DataFrame contains a 'rating' column with some missing values represented as `np.nan`. The goal is to replace these missing values with the average rating of the column. Which of the following pandas operations correctly performs this imputation?

Accepted Answer

df['rating'].fillna(df['rating'].mean())

Answer

The `fillna()` method is the standard way to replace missing values (NA/NaN) in pandas. By passing `df['rating'].mean()` as the argument, it calculates the mean of the non-missing values in the 'rating' column and uses that result to fill all the `np.nan` entries.

Data Science with Python Certification Practice Test

Data Science with Python Certification Practice Test

Data Science with Python Data Cleaning and Preparation Questions and Answers