Feature Selection in ML

Feature selection changes the accuracy of a model less than the order in which it is applied. A model trained on features chosen before the train/test split reports higher accuracy than the same model tested honestly. The difference between the two figures is what the article is all about.

Feature selection is the process of choosing which input variables a machine learning model used for training, based on their relevance to the target variable. All other features are removed before training starts.

The article explores three types of feature selection techniques, when you should use one over the other, the leakage mistake that results in inflated accuracies, when feature selection is useless, and includes a working Python code example.

What Is Feature Selection?

Feature selection refers to the task of picking a subset of important input features from the data set, while eliminating those that may cause noise or redundancy in the modeling.

Feature selection represents one of the stages of feature engineering. Feature engineering refers to all the actions that need to be taken when preparing data for modeling. These actions include the creation, transformation, and selection of new features. Feature selection concerns itself only with the action of selection.

Feature selection and feature extraction should not be confused. While feature selection involves retaining a subset of the original features, feature extraction involves creating new features through transformations or combinations of the existing features. Principal Component Analysis (PCA) is a feature extraction technique and not a feature selection technique since PCA creates new components and does not retain any of the original columns.

What Counts as a Feature?

A feature is an attribute of data points used to construct models. Examples of features in a dataset of housing are square footage, number of rooms, and year built. The classification of features is done through:

Feature type	Definition	Example
Numerical	A feature with a quantifiable, ordered value	Age, price, square footage
Categorical	A feature with a non-numerical, unordered value	City, job title, product category

The target variable is the output the model is trained to predict. Feature selection identifies which input features have the strongest relationship with the target variable.

Why Feature Selection Matters

Feature selection produces five measurable effects on a model. Each effect is a trade, not a guaranteed gain.

Effect	Mechanism	Condition for benefit
Higher accuracy	Removes variables that add noise to the training signal	Only when removed variables were genuinely uninformative
Lower overfitting	Fewer parameters reduce the model’s capacity to memorize training data	Most useful on high-dimensional, low-sample datasets
Shorter training time	Fewer input columns reduce computation per training step	Scales with the number of features removed
Lower inference cost	Fewer features reduce the data pipeline required at prediction time	Matters most in production systems with per-feature retrieval cost
Greater interpretability	Fewer variables make model behavior easier to trace	Most relevant for regulated or audited use cases

Feature selection is not an assurance of accuracy improvement. Deleting any feature containing genuine information decreases the accuracy. It all depends on being able to identify the features that do not contain useful information.

The Curse of Dimensionality

The Curse of Dimensionality is a phenomenon of decline in model performance due to an increasing ratio of input dimensions to the number of training samples, as data points become more sparse in higher-dimensional space.

As we move into more dimensions, the feature space volume increases exponentially while the number of data points remains constant. For instance, the data set of 100 observations with 5 dimensions has data points at an interval of approximately 0.4 on each dimension according to the usual distance formula 1/N^1/d where N is the total number of observations and d is the number of dimensions. Similarly, the same number of data points with 50 features has almost empty feature space, as the interval between each data point comes down to an approximate distance of 0.09 on each dimension. 

Feature Selection vs. Feature Extraction vs. Dimensionality Reduction

Property	Feature selection	Feature extraction	Dimensionality reduction
Output	A subset of original features	New, transformed features	Either selection or extraction
Modifies original values	No	Yes	Depends on method
Interpretability	High — original features remain identifiable	Lower — new features are combinations	Depends on method
Example methods	Filter, wrapper, embedded methods	PCA, Linear Discriminant Analysis (LDA), autoencoders	Includes both categories above
Relationship	A category of dimensionality reduction	A category of dimensionality reduction	The umbrella term for both categories

Feature selection is a subclass of dimensionality reduction. Feature extraction is another subclass. Dimensionality reduction refers to any technique that reduces the number of input dimensions without consideration for whether the original feature space is preserved or altered.

The Three Types of Feature Selection Methods

Feature selection methods are grouped into three categories based on how they interact with the machine learning model: filter, wrapper, and embedded methods.

Method type	How it works	Uses the model during selection	Relative speed	Best suited for
Filter	Scores each feature using a statistical test against the target variable	No	Fast	Large datasets, initial screening
Wrapper	Trains and evaluates the model on different feature subsets	Yes	Slow	Small to mid-sized feature spaces
Embedded	Selects features as part of the model’s own training process	Yes, intrinsically	Moderate	Regularized linear models, tree-based models

Filter Methods

Filter approaches involve the selection of features based on their score obtained individually using statistical measures for their relevance to the target variable; no machine learning model is trained. Because filter approaches do not rely on any particular model, the resulting ranking holds for any model.

Technique	Statistical basis	Applies to
Information gain	Reduction in entropy when a feature is known	Categorical target
Mutual information	Shared information between two variables	Numerical or categorical
Chi-square test	Association between two categorical variables	Categorical input and target
Fisher’s score	Class separability by feature	Classification tasks
Pearson’s correlation coefficient	Linear relationship, ranging from -1 to 1	Continuous input and target
Variance threshold	Removes features below a set variance value	Any numerical feature
Missing value ratio	Percentage of missing entries per feature	Any feature type
ANOVA	Whether group means differ significantly	Categorical input, numerical target

Filter approaches cannot eliminate multicollinearity. In cases where two variables have high scores on the target feature, they may contain identical information. There is a need to apply an additional approach in order to eliminate correlated variables.

Wrapper Methods

Wrapper methods train models on various sets of features and retain the one that gives the highest validation accuracy. The wrapper method evaluates the feature set in terms of its accuracy, while filter methods do not.

Technique	Process	Stopping condition
Forward selection	Starts with zero features, adds one at a time	Stops when adding a feature no longer improves performance
Backward elimination	Starts with all features, removes one at a time	Stops when removing a feature reduces performance
Recursive Feature Elimination (RFE)	Trains the model, ranks features by importance, removes the lowest-ranked feature, repeats	Stops at a preset number of features
RFE with cross-validation (RFECV)	Runs RFE across multiple validation folds	Stops at the feature count with the highest average cross-validated score
Exhaustive selection	Tests every possible feature combination	Stops after all combinations are scored

Exhaustive selection checks all the 2ⁿ − 1 possible combinations of n features. If there are 20 features, the number of combinations would be 1,048,575. The above process becomes too costly to use beyond small numbers of features, and forward, backward, and recursive selections become common methods.

The Boruta algorithm is a wrapper algorithm that works on the basis of feature importance, in which it compares every feature with its randomized version(shadow feature). A feature is selected if it is better than its shadow feature.

Embedded Methods

The embedded method is one where the selection of features occurs during the training of the model, based on the optimization mechanism of the model itself.

Technique	Mechanism	Output
LASSO regression (L1 regularization)	Adds a penalty proportional to the absolute value of each coefficient	Coefficients of irrelevant features shrink to exactly zero
Elastic net	Combines L1 and L2 penalties	Selects features while retaining groups of correlated ones
Random forest importance	Measures the average reduction in impurity a feature produces across all trees	A ranked importance score per feature
Gradient boosting importance	Measures the reduction in prediction error a feature produces across boosting iterations	A ranked importance score per feature

The ridge regression technique (L2 regularization) is not a feature selection approach. L2 regularization makes the model coefficients smaller without setting them to zero; thus, all of the features are included in the model. The only way that we can obtain coefficients equal to zero and thus perform feature selection is through L1 regularization.

Hybrid and Unsupervised Approaches

Hybrid approaches use a filtering phase, which filters out irrelevant features, followed by a wrapping or embedded phase to select the subset from the filtered feature pool, thus cutting down the cost associated with performing a wrapper approach on all the features.

Unsupervised feature selection is applied where there is no target variable. PCA, ICA, and autoencoders are employed in this case. The three approaches mentioned above are feature extraction methods, not feature selection methods, since they generate new features and do not select a subset of the initial features.

How to Choose a Feature Selection Method

The correct method depends on the data type of the input and output variables.

Input type	Output type	Problem type	Recommended method
Numerical	Numerical	Regression	Pearson’s correlation coefficient
Numerical	Categorical	Classification	ANOVA (linear), Kendall’s rank correlation (nonlinear)
Categorical	Numerical	Regression	Correlation methods that support categorical variables
Categorical	Categorical	Classification	Chi-square test, information gain

Three additional factors determine method choice:

1. Size of dataset. Filter approaches can handle large datasets. Wrapper approaches require higher computational cost as the number of features increases since each new feature causes an increase in the number of subsets that need to be checked.
2. Type of model. Tree-based models generate feature importance rankings as a part of model fitting, hence the use of embedded approach is more logical. In case of distance-based models like k-Nearest Neighbors, the use of filter or wrapper approaches is more appropriate.
3. Need for interpretability. A feature may have an importance score in a trained model while never being considered by any feature selection technique. Wrapper and embedded approaches are more difficult to explain to an outsider as they rely on the performance of the model.

The Mistake That Breaks Everything: Selecting Before You Split

Performing feature selection on the whole dataset prior to the split of it into the training and test set results in the test set impacting the selected features, thus generating a biased accuracy figure that does not reflect the true accuracy figure for real unseen data.

The Mechanism: All the feature selection algorithms, among them the filter ones, compute certain metrics such as the correlation and the information gain based on all available rows. When the test set impacts the computation of those metrics, the selected features depend on the specific data on which the model will be scored.

The correct sequence is:

1. Divide the dataset into training and testing datasets initially.
2. Use the training dataset only to fit the feature selection algorithm.
3. Use the selected features to both train and test datasets.
4. Train the model using the training dataset with selected features.
5. Test the model using the test dataset.

Nested cross-validation applies this same sequence within each cross-validation fold, so that feature selection is refit independently for every fold rather than once on the full dataset. RFECV and Pipeline in scikit-learn implement this automatically when feature selection is placed inside the pipeline object before cross-validation is run.

When Feature Selection Provides No Benefit

Feature selection does not improve every model. Three conditions determine when it adds no measurable value:

1. Regularized linear models. LASSO and elastic nets select features as part of training. Performing any additional filter or wrapper step beforehand results in redundant computation by the algorithm itself.
2. Gradient-boosted tree models. XGBoost, LightGBM, and CatBoost automatically give near zero importance scores to the unimportant features during training for data sets having a modest number of features. Any pre-processing step for such algorithms often results in no difference in accuracy whatsoever.
3. Low feature counts relative to sample size. In the case where the number of features is few in comparison to the number of training samples, there is no curse of dimensionality and the overhead of feature selection exceeds its benefits.

The selection of features yields the greatest quantifiable gains in the case of datasets that are highly dimensional with very few sample points, when distance-based algorithms or linear models are not regularized. Feature selection is applicable to genomic data, as well as in the classification of text when using a bag-of-words model without further feature engineering.

Feature Selection in Python: A Working Example

The following example applies VarianceThreshold, SelectKBest, and RFECV inside a single scikit-learn Pipeline, so that feature selection is refit within each cross-validation fold rather than applied once to the full dataset.

from sklearn.pipeline import Pipeline

from sklearn.feature_selection import VarianceThreshold, SelectKBest, f_classif, RFECV

from sklearn.ensemble import RandomForestClassifier

from sklearn.model_selection import StratifiedKFold, cross_val_score

# Step 1: remove near-zero-variance features

variance_filter = VarianceThreshold(threshold=0.01)

# Step 2: filter method — keep the top 20 features by ANOVA F-value

filter_step = SelectKBest(score_func=f_classif, k=20)

# Step 3: wrapper method — recursive elimination with cross-validation

estimator = RandomForestClassifier(n_estimators=200, random_state=42)

wrapper_step = RFECV(estimator=estimator, step=1, cv=5, scoring="accuracy")

pipeline = Pipeline([

    ("variance_filter", variance_filter),

    ("filter", filter_step),

    ("wrapper", wrapper_step),

    ("model", estimator),

])

# Feature selection is refit inside each fold — not on the full dataset

cv = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)

scores = cross_val_score(pipeline, X, y, cv=cv, scoring="accuracy")

print(f"Mean cross-validated accuracy: {scores.mean():.3f}")

Placing SelectKBest and RFECV inside the Pipeline object, and calling cross_val_score on the full pipeline, ensures scikit-learn refits both steps using only the training fold at each iteration. Calling SelectKBest or RFECV on the full dataset before cross_val_score reproduces the leakage error described in the previous section.

How Many Features Should You Keep?

The number of features to keep is determined by the point at which cross-validated accuracy stops improving, identified using RFECV‘s per-step validation scores rather than a fixed target count.

RFECV records the cross-validated score at each feature count as it eliminates features one at a time. Plotting these scores produces a curve that typically rises, plateaus, and then declines as informative features are removed alongside noise. The recommended feature count is the smallest number of features within one standard error of the peak score, rather than the exact peak, because the exact peak can reflect noise specific to the validation folds used.

Feature Importance Is Not Feature Selection

The importance of a feature refers to a score that indicates how important the particular feature is for predictions made by the trained model. Feature selection, on the other hand, is a process where the choice of the selected features for the model is made.

A feature may have an importance score in a certain trained model while being not at all considered by any of the feature selection techniques, and a feature that is filtered out by a filter technique never gets an importance score because it is not even included in the model training process.

Method	Measures	Computed
SHAP values	Each feature’s individual contribution to a specific prediction	After training, using a game-theoretic allocation
Permutation importance	The drop in model accuracy when a feature’s values are randomly shuffled	After training, by re-scoring the model repeatedly
Gini importance (tree models)	Average impurity reduction attributable to a feature across all trees	During training, as a byproduct

Gini importance suffers from an inherent bias in favor of categorical variables with higher cardinality since the higher the cardinality of a feature, the more chances there will be for generating a split which results in a lower degree of impurity regardless of whether or not the feature has any relationship with the target.

Handling Correlated and Redundant Features

Highly correlated features carry similar information, and including two of them in a model increases the variance of the coefficient estimates without improving predictive power. 

The variance inflation factor (VIF) shows what increase in variance is caused by the correlation of the feature with all other features in the model.

VIF value	Interpretation
1	No correlation with other features
1–5	Moderate correlation, generally acceptable
5–10	High correlation, review required
Above 10	Severe multicollinearity, feature removal recommended

One such heuristic drops one feature of each pair whose correlation coefficient ranges between 0.85 and 0.90, retaining the one having a stronger association with the target variable.

Maximum Relevance Minimum Redundancy (mRMR) is an example of a filter approach where the selection of the features involves the simultaneous maximization of the relevance of the selected features to the target variable and minimization of redundancy with previously selected features, instead of individual ranking of the features. 

A correlated feature is not always a redundant one. Two features that are correlated with each other can both be relevant to the target variable, and dropping one without verifying its relevance to the target variable may lower the prediction accuracy of the model.

Common Feature Selection Mistakes

1. Feature selection performed prior to splitting the data set. Creates an inflated accuracy estimate, as explained above.
2. Relying on a single feature selection without assessing its stability.Running a feature selection wrapper or filter again on different subsets of data or cross-validation folds results in a different feature subset being selected each time. If the feature subset changes significantly in each iteration, then this implies the feature selection is noise-sensitive rather than stable.
3. Assessing Gini importance of high cardinality categorical features without adjustment. Creates an inflated importance measure for high cardinality categorical features.
4. Performing feature selection on time-series data without taking into account its temporal nature. Cross-validation randomly shuffles the data, which makes it possible for future-time information to leak during the feature selection procedure. Time-series feature selection needs a time-sorted validation split.
5. Deleting a protected attribute without testing for the presence of proxy variables. Deletion of a feature such as ZIP code will not delete the information if there is a correlated variable such as the average income per neighborhood.

FAQs

Summary Comparison Table

Method type	Model-aware	Relative cost	Handles feature interactions	Common techniques
Filter	No	Low	No	Chi-square, ANOVA, correlation, mutual information
Wrapper	Yes	High	Yes	Forward selection, backward elimination, RFE, RFECV
Embedded	Yes (intrinsic)	Moderate	Partial	LASSO, elastic net, random forest importance

Whether an accuracy figure presented is reliable depends on the order of operations. An approach to feature selection nested in a cross validation cycle and relying on the training set only in every repetition gives the accuracy that the classifier can get on new data. Otherwise, it won’t.