DSE Feature Engineering and Selection 5 — Questions and Answers
Question 1: What does 'feature importance' from a gradient boosting model (e.g., XGBoost) typically measure?
- The p-value of each feature in a linear regression
- The average gain in model performance when a feature is used in a split across all trees (Correct answer)
- The correlation of each feature with the residuals
- The number of times each feature appears in the training data
Correct answer: The average gain in model performance when a feature is used in a split across all trees
XGBoost's gain-based feature importance averages the improvement in the loss function brought by each feature across all splits where it is used.
Question 2: Which of the following correctly describes 'entity embeddings' as a feature engineering technique?
- Applying PCA to all continuous features jointly
- Learning dense low-dimensional vector representations for categorical features via a neural network (Correct answer)
- Encoding entities as one-hot vectors in a sparse matrix
- Hashing entity IDs to a fixed-size bucket array
Correct answer: Learning dense low-dimensional vector representations for categorical features via a neural network
Entity embeddings train a neural network to map each category to a dense vector, capturing semantic relationships that one-hot encoding cannot.
Question 3: In geospatial feature engineering, what is the purpose of computing Haversine distance as a derived feature?
- To encode city names as numeric identifiers
- To compute the great-circle distance between two latitude/longitude points on a sphere (Correct answer)
- To normalize latitude and longitude to the range [0,1]
- To cluster geographic regions into bins
Correct answer: To compute the great-circle distance between two latitude/longitude points on a sphere
The Haversine formula calculates the shortest distance over the Earth's surface between two coordinate pairs, creating a meaningful numeric proximity feature.
Question 4: What is 'Boruta' and how does it differ from standard feature importance thresholding?
- A normalization technique that standardizes features to unit variance
- A feature selection algorithm that compares each feature's importance against shadow (permuted) features to determine statistical significance (Correct answer)
- A wrapper method that trains one model per feature
- An embedded method that adds L1 penalty to eliminate features
Correct answer: A feature selection algorithm that compares each feature's importance against shadow (permuted) features to determine statistical significance
Boruta creates randomized copies of features (shadows), trains a Random Forest, and accepts only features that consistently outperform the best shadow feature.
Question 5: When engineering date/time features, which decomposition is most useful for capturing weekly seasonality in a demand forecasting model?
- Year and month extraction
- Day of week extraction (Correct answer)
- Hour of day extraction
- Unix timestamp normalization
Correct answer: Day of week extraction
Day of week captures the recurring weekly pattern in demand (e.g., weekday vs. weekend effects) that is typically the strongest seasonal cycle in retail data.
Question 6: What is the purpose of 'clipping' (winsorization) as a feature preprocessing step?
- To remove all outliers from the dataset permanently
- To cap extreme values at specified percentile thresholds, reducing the influence of outliers without deleting rows (Correct answer)
- To scale features to a fixed range like [0,1]
- To replace missing values with boundary values
Correct answer: To cap extreme values at specified percentile thresholds, reducing the influence of outliers without deleting rows
Winsorization sets values below the lower percentile to that lower bound and values above the upper percentile to that upper bound, limiting outlier influence while retaining all rows.
Question 7: Which method is best suited for selecting features when the relationship between features and the target is highly non-linear and non-monotonic?
- Pearson correlation filter
- Chi-squared test for continuous targets
- Mutual information-based filter (Correct answer)
- Variance threshold filter
Correct answer: Mutual information-based filter
Mutual information captures any statistical dependency between a feature and the target, including non-linear and non-monotonic relationships, unlike correlation which only detects linear ones.
DSE Feature Engineering and Selection 5
What does 'feature importance' from a gradient boosting model (e.g., XGBoost) typically measure?