Machine Learning Feature Engineering 5 — Questions and Answers
Question 1: What is 'feature agglomeration' in scikit-learn?
- Selecting the top-k features by mutual information
- Merging similar features into groups using hierarchical clustering (Correct answer)
- Creating polynomial combinations of existing features
- Removing features with missing values above a threshold
Correct answer: Merging similar features into groups using hierarchical clustering
FeatureAgglomeration applies hierarchical clustering to group correlated features into clusters and replaces each cluster with its centroid value.
Question 2: Which approach is most effective for handling cyclical features like 'hour of day' or 'month of year'?
- Normalizing them to [0,1]
- Encoding them as sine and cosine components (Correct answer)
- Treating them as ordinal integers
- One-hot encoding each value
Correct answer: Encoding them as sine and cosine components
Sine and cosine encoding preserves the cyclical nature so that hour 23 and hour 0 are close together in the encoded space.
Question 3: What is the 'curse of dimensionality' effect on distance-based models when many features are added?
- Training becomes faster as more features provide signal
- Distances between points become increasingly similar, making nearest-neighbor methods unreliable (Correct answer)
- The model becomes more interpretable
- Overfitting decreases because more variance is explained
Correct answer: Distances between points become increasingly similar, making nearest-neighbor methods unreliable
In high-dimensional spaces, pairwise distances concentrate around the same value, reducing the discriminative power of distance-based algorithms like k-NN.
Question 4: When applying PCA for dimensionality reduction before classification, on which dataset should the PCA be fit?
- Both training and test sets combined
- The training set only, then applied to the test set (Correct answer)
- The test set only
- A separate validation set
Correct answer: The training set only, then applied to the test set
Fitting PCA only on training data prevents test-set information from influencing the transformation, which would constitute data leakage.
Question 5: What is 'weight of evidence' (WoE) encoding most commonly used for?
- Regularizing tree-based models
- Encoding categorical variables for binary classification, especially in credit scoring (Correct answer)
- Upsampling minority class samples
- Scaling numerical features to zero mean
Correct answer: Encoding categorical variables for binary classification, especially in credit scoring
WoE encodes categories based on the log ratio of the proportion of events to non-events, which is particularly useful in logistic regression for credit risk models.
Question 6: Which of the following best describes 'feature importance' from a Random Forest model?
- The correlation coefficient of a feature with the target variable
- The average reduction in impurity (e.g., Gini) across all trees when a feature is used to split (Correct answer)
- The p-value of a feature in a linear regression
- The number of times a feature appears in the training data
Correct answer: The average reduction in impurity (e.g., Gini) across all trees when a feature is used to split
Random Forest feature importance measures how much each feature decreases weighted impurity across all splits in all trees, averaged over the ensemble.
Question 7: What does 'SMOTE' (Synthetic Minority Oversampling Technique) do during feature engineering for imbalanced datasets?
- Removes duplicate majority-class samples
- Generates synthetic minority-class samples by interpolating between existing minority samples (Correct answer)
- Applies class weights to the loss function
- Performs random undersampling of the majority class
Correct answer: Generates synthetic minority-class samples by interpolating between existing minority samples
SMOTE creates new minority-class examples by interpolating along line segments joining a minority sample and its nearest minority neighbors in feature space.
Machine Learning Feature Engineering 5
What is 'feature agglomeration' in scikit-learn?