A machine learning model fed the values “blue = 2” and “red = 1” treats blue as numerically greater than red. One-hot encoding eliminates this incorrect numerical relationship by assigning each category a separate binary column. This post discusses the way one-hot encoding works, its implementation in Python, and situations when another encoding method works better.
What Will I Learn?
What Is One-Hot Encoding?
One-hot encoding is a data preprocessing method that converts a categorical variable into a set of binary columns, one per category, marking presence with 1 and absence with 0.
A Color column with three categories — Red, Blue, and Green — becomes three columns: Color_Red, Color_Blue, and Color_Green.
| Color | Color_Red | Color_Blue | Color_Green |
|---|---|---|---|
| Red | 1 | 0 | 0 |
| Blue | 0 | 1 | 0 |
| Green | 0 | 0 | 1 |
Machine learning algorithms, such as linear and logistic regression, neural networks, and so on, need numeric inputs. One-hot encoding provides this without establishing a numeric connection where there is none.
Why You Can’t Just Use Label Encoding Instead
Label encoding labels each class with an integer, creating a forced ranking between classes that does not actually exist. There are two forms of categorical data, namely, nominal data, where an order between data points does not exist, and ordinal data, where an order does exist.
Color, city, and product type belong to the former; nominal data has no logical ordering between data points – there is no way of knowing whether blue is ‘greater’ than red. Size of shirts (small, medium, large) and customer satisfaction (poor, fair, good) are examples of ordinal data where an ordering can be identified.
When performing label encoding on nominal data – for instance, assigning Apple = 1, Chicken = 2, Broccoli = 3 – a numeric weight is assigned to each data point. When the algorithm is doing a calculation involving distance or averaging between data points, the algorithm will consider chicken to be twice as much as apple, and the mid-point between apple and broccoli to be chicken, even though it does not make sense in the real world.
Machine Learning Course
Average time: 5 month(s)
Skills you’ll build: Python, Scikit-learn, Supervised & Unsupervised Learning, Feature Engineering, Model Deployment, and more..
How One-Hot Encoding Works, Step by Step
One-hot encoding entails the identification of unique values in a column of data and the creation of a new column for each unique value, such that each row takes a value of 1 in one column only and 0s in the rest.
There are three steps involved:
- Identify all unique categories in the column of data. In a Fruit column that has Apple, Mango, and Orange, there are three unique categories.
- Make a new column for each unique category. There will be three new columns: Fruit_Apple, Fruit_Mango, Fruit_Orange.
- For each row, make the column that matches the unique category of that row have a value of 1 and the rest have zeros.
| Fruit | Price | Fruit_Apple | Fruit_Mango | Fruit_Orange |
|---|---|---|---|---|
| Apple | 5 | 1 | 0 | 0 |
| Mango | 10 | 0 | 1 | 0 |
| Apple | 15 | 1 | 0 | 0 |
A categorical column with n unique values always produces n new binary columns. A Country column with 50 unique values produces 50 new columns, one per country.
One-Hot Encoding in Python: Pandas vs. Scikit-learn
Although both scikit-learn and pandas do one hot encoding, they do it for different purposes; whereas pandas.get_dummies() is made for fast, one-off transformation of the DataFrame, sklearn.OneHotEncoder is made for transforming machine learning pipeline where the same transformation will need to be done on new data.
Using pandas.get_dummies()
pandas.get_dummies() converts every categorical column passed to it into binary columns in a single function call.
import pandas as pd
data = {
'Employee_ID': [10, 20, 15, 25, 30],
'Gender': ['M', 'F', 'F', 'M', 'F'],
'Department': ['Sales', 'IT', 'Sales', 'HR', 'IT']
}
df = pd.DataFrame(data)
encoded_df = pd.get_dummies(df, columns=['Gender', 'Department'], drop_first=True)
print(encoded_df)It is efficient in exploratory analyses and one-time transformations. The fact that it doesn’t store the categories it encounters means that it can’t reliably apply the same transformation to a different dataset in the future.
Using sklearn.OneHotEncoder()
sklearn.OneHotEncoder fits on a training set, stores the categories it learned, and applies that exact same mapping to any future data.
import pandas as pd
from sklearn.preprocessing import OneHotEncoder
data = {
'Gender': ['M', 'F', 'F', 'M', 'F'],
'Department': ['Sales', 'IT', 'Sales', 'HR', 'IT']
}
df = pd.DataFrame(data)
encoder = OneHotEncoder(drop='first', sparse_output=False, handle_unknown='ignore')
encoded_array = encoder.fit_transform(df[['Gender', 'Department']])
encoded_df = pd.DataFrame(
encoded_array,
columns=encoder.get_feature_names_out(['Gender', 'Department'])
)
print(encoded_df)Scikit-learn version 1.2 renamed the sparse parameter to sparse_output. Code written against OneHotEncoder(sparse=False) will raise a deprecation warning or error on current scikit-learn releases; use sparse_output=False instead.
| Task | pandas.get_dummies() | sklearn.OneHotEncoder |
|---|---|---|
| One-time transformation of a full dataset | Fits the task | Works, but adds unnecessary steps |
| Reusable encoding for a train/test split | Not designed for this | Fits the task |
| Handling unseen categories in new data | No built-in handling | Built-in via handle_unknown |
| Integration into a scikit-learn Pipeline | Not supported directly | Supported directly |
Avoiding the Dummy Variable Trap
The dummy variable trap occurs when the full set of one-hot encoded columns for a variable is perfectly predictable from the remaining columns, creating multicollinearity.
A Gender column with two categories — Male and Female — produces two columns after one-hot encoding: Gender_Male and Gender_Female. Every row where Gender_Male is 0 has Gender_Female equal to 1, and every row where Gender_Male is 1 has Gender_Female equal to 0. The second column adds no new information once the first column is known.
Linear regression and logistic regression compute coefficients by inverting a matrix built from the input features. Perfectly correlated columns make that matrix impossible to invert reliably, which produces unstable or undefined coefficients. Setting drop=’first’ in sklearn.OneHotEncoder or drop_first=True in pandas.get_dummies() removes one column per categorical variable and eliminates the redundancy. The dropped category is not lost — a row where every remaining Gender column reads 0 is understood by the model to represent that dropped category.
Decision trees and random forests do not make use of any matrix inverse computations and are not impacted by the presence of the redundancy to the same extent. The removal of the first column is not mandatory for all algorithms but only linear models.
What Happens When a New Category Shows Up at Prediction Time?
A one-hot encoder trained on fixed categories errors by default on an unseen category; handle_unknown=’ignore’ in sklearn.OneHotEncoder avoids this by encoding it as all zeros.
The model that is built on the City feature where only New York, Chicago, and Boston appear will have no City feature when Miami appears in the production data. By default, the OneHotEncoder from scikit-learn raises the ValueError when it occurs.
encoder = OneHotEncoder(handle_unknown=’ignore’, sparse_output=False)
With handle_unknown=’ignore’, a row containing an unseen category receives a value of 0 in every column for that variable. The model still receives valid numeric input, though it has no information distinguishing that new category from any category it has already learned. pandas.get_dummies() has no equivalent parameter; a column encoded with get_dummies() on new data must be manually re-aligned to match the training set’s columns using DataFrame.reindex().
Machine Learning Course
Average time: 5 month(s)
Skills you’ll build: Python, Scikit-learn, Supervised & Unsupervised Learning, Feature Engineering, Model Deployment, and more..
When One-Hot Encoding Works Well — and When It Doesn’t
Advantages
- Eliminates incorrect ordinal relations among categories where there is no natural ordering.
- Generates numerical input suitable for any common machine learning algorithm.
- Retains all information from each category intact.
Limitations
- Adds a column for each unique category that exists within a column, thereby increasing the dimensionality of the dataset.
- Creates a sparse dataset, or a dataset that contains mostly zeros, whenever a column is categorical and has many unique categories.
- May adversely affect the model’s accuracy and increase the training time on columns with high cardinality.
Better Alternatives for High-Cardinality Data
If a categorical attribute has more than 15-20 unique values, then the attribute is said to be high-cardinality. One hot encoding results in a huge and sparse feature space.
| Method | How It Works | Best Fit |
|---|---|---|
| Target encoding | Replaces each category with the average target value for that category | Regression and classification tasks with high-cardinality features |
| Frequency encoding | Replaces each category with the count or proportion of rows containing that category | Tree-based models, fast baseline encoding |
| Hash encoding | Maps each category to a fixed number of columns using a hash function | Very high-cardinality features, such as user IDs |
| Embeddings | Learns a dense numeric vector for each category during model training | Deep learning models, especially with millions of unique categories |
A ZIP Code column containing 40,000 unique values, one-hot encoded, produces 40,000 new columns. Target encoding or hashing produces the same predictive information in a fraction of the columns.
Does It Matter for Tree-Based Models Too?
One-hot encoding is not needed in tree-based algorithms like decision trees, random forests, and gradient boosting, as they do splitting on the basis of thresholding and not numeric magnitude or distance.
A linear model calculates a weighted sum of all the features that are used, which means that any numeric ordering in the input due to any form of encoding will matter. However, tree-based models make yes or no questions regarding each of the input features; for instance, “Is this particular row’s value equal to 3?”. The tree algorithm will be able to split on a categorical feature that has been label encoded without interpreting the numeric order incorrectly.
A high cardinality categorical feature will hamper the performance of a tree-based model even when it has been one-hot encoded beforehand since the wide and sparse feature space generated from this operation forces the tree to do many splits in order to capture information that could have been captured in just a single column using label encoding or target encoding.
One-Hot Encoding Beyond Tables: NLP and Deep Learning
In natural language processing, one hot encoding works as assigning a binary vector having only one 1 at the index of a particular word and zero at all the other indices for each word of the vocabulary.
Thus, a vocabulary of 10,000 words will result in a vector with 10,000 elements for each word with exactly one element being equal to 1. One hot encoding requires linear space and does not encode any similarity between words; that is, “happy” and “joyful” would get completely dissimilar vectors. Word embedding techniques, used by transformer-based language models, use dense, learned vector representations instead of one-hot vectors.
Some deep learning libraries offer direct functionality to generate one hot encoded vectors:
- TensorFlow: tf.one_hot(indices, depth)
- PyTorch: torch.nn.functional.one_hot(tensor, num_classes)
These two functions accept an input tensor of integer class indices along with the number of classes as parameters and output a tensor having a binary vector for each input index. Such functions are frequently employed in transforming classification labels for training purposes.
Frequently Asked Questions
Q1. Is one-hot encoding the same as dummy encoding?
Ans. One-hot encoding uses one column per category, whereas dummy encoding uses one less column per category by removing a reference category; use of drop_first=True in pandas.get_dummies() gives us dummy encoding.
Q2. Does one-hot encoding work with missing values?
Ans. One-hot encoding does not handle missing values automatically; they must be filled or given their own category before encoding, or the encoder will raise an error or drop the affected rows.
Q3. Can one-hot encoding cause overfitting?
Ans. One-hot encoding a high-cardinality feature can cause overfitting, because the resulting sparse columns allow a model to memorize rare category combinations instead of learning generalizable patterns.
Q4. Should I one-hot encode before or after splitting data into train and test sets?
Ans. One hot encoding should be fitted only on the training data and transformed on the test data using transform(), not fit_transform() to ensure that the distribution of categories of the test data doesn’t influence the training phase.
Q5. What is the difference between one-hot encoding and ordinal encoding?
Ans. One hot encoding creates a separate binary variable for each category with no ordering while ordinal encoding creates a single numeric variable for each category. Ordinal encoding should only be performed when there is a rank order in categories such as Low, Medium, and High.
One-hot encoding serves a certain purpose: representing the nominal data in numeric form without making up the order, which does not exist. In the case of low cardinality nominal attributes fed into a linear model, one-hot encoding is the method of choice. In other cases, target encoding, frequency encoding or even native category support in a model can generate a more compact and accurate set of features.
