Data Preprocessing

Most tutorials tell you that data preprocessing is a four-step process. It is not. It is a series of decisions – getting any one wrong at any point in the sequence can poison your model without emitting even a single warning signal.

As stated by the 2020 State of Data Science report by Anaconda, data scientists typically allocate 45% of their total project time towards preparing the data to be modeled – that is, loading and cleaning the data. The model is easy. The data preparation process makes or breaks projects. The model is easy. The data preparation process makes or breaks projects.

This guide provides all the necessary details on the entire process, from start to finish.

What Is Data Preprocessing — And Why It Can’t Be Skipped

Data preprocessing involves cleaning, integration, transformation, and reduction of the data to make it useful for the inputting of a machine learning model or pipeline.

The raw data is almost always unusable in its form. The data is incomplete, contains duplicates, inconsistency of formatting, and features vary drastically in scale. One database may record the age of a customer as “28,” while another database may record “twenty-eight.” The value of “9,999” for temperature indicates a failure in the sensor. An entire column filled with 40 percent missing values.

Put this data in your machine learning algorithm without cleaning and pre-processing? Sure, you’ll get an answer. But it won’t really make sense.

The problem with bad data preprocessing is that it does not always break down in the most dramatic way possible. In fact, sometimes it creates a very successful model which fails when exposed to real-world inputs.

The 4 Steps of Data Preprocessing — In the Right Order

The four steps are: clean, integrate, transform, reduce. Every major source agrees on these steps. What they don’t tell you is that sequence matters — and that one action has to happen before any of them.

Split your data before you preprocess it.

No choice. When you fit a StandardScaler to your entire dataset before splitting it into train and test sets, you have already used your test data’s information in your training process. Your model has effectively seen the answers before the test. You will get results that are better than they should be. This is called data leakage, and it is a common reason why models fail in production despite performing well in development. A peer-reviewed paper from 2023 showed that it was a prevalent issue for science using machine learning models, impacting hundreds of academic papers in various fields.

The correct order:

1. Split your data into train and test sets
2. Preprocess only your training set
3. Apply the same preprocessing pipeline without refitting to your test set

The rest of this article assumes that you have done this.

Step 1 — Data Cleaning: Fixing What’s Broken

Data cleansing involves identifying errors in your dataset and correcting them. Three common errors recur consistently: missing values, outliers, and duplication.

Handling Missing Values – Choosing the Right Strategy

It is inevitable to encounter missing values; however, the replacement is not random.

It depends on the reason for missing values and how your data is distributed.

Situation	Method	sklearn Class
Symmetric distribution, no major outliers	Mean imputation	SimpleImputer(strategy=’mean’)
Skewed data or outliers present	Median imputation	SimpleImputer(strategy=’median’)
Categorical column	Mode imputation	SimpleImputer(strategy=’most_frequent’)
Missingness relates to other features	KNN imputation	KNNImputer(n_neighbors=5)
Under 5% of rows affected	Row deletion	df.dropna()

Though KNN Imputer is slow, it’s smart. Instead of taking the average throughout the dataset, it considers k-nearest records and imputes the missing value accordingly. In cases where missing values patterns are more complicated and we’re dealing with smaller data sets, the extra computation is worth it.

python
from sklearn.impute import SimpleImputer, KNNImputer

# Median imputation
median_imputer = SimpleImputer(strategy='median')
X_train_imputed = median_imputer.fit_transform(X_train)
X_test_imputed = median_imputer.transform(X_test)  # transform only -- no re-fitting

# KNN imputation
knn_imputer = KNNImputer(n_neighbors=5)
X_train_knn = knn_imputer.fit_transform(X_train)
X_test_knn = knn_imputer.transform(X_test)

transform() on the test set. Not fit_transform(). That distinction is the whole point.

Detecting and Removing Outliers

There are two approaches for most cases.

The Z-Score approach is applicable to data that is normally distributed. Any data point beyond 3 standard deviations is considered an outlier.

For skewed data, the IQR (Interquartile Range) approach can be used. Data points below Q1 – 1.5×IQR or above Q3 + 1.5×IQR are outliers.

python
from scipy import stats

# Z-Score
z_scores = stats.zscore(df['price'])
df_clean = df[abs(z_scores) < 3]

# IQR
Q1 = df['price'].quantile(0.25)
Q3 = df['price'].quantile(0.75)
IQR = Q3 - Q1
df_clean = df[
    (df['price'] >= Q1 - 1.5 * IQR) &
    (df['price'] <= Q3 + 1.5 * IQR)
]

A tip that is usually overlooked in most guides is to avoid automatically eliminating outliers from any datasets used in finance or medicine. For example, a transaction worth $50,000 may be considered a statistical outlier, yet it might actually be the transaction made by your best paying client.

Removing Duplicates

python
df = df.drop_duplicates()

That handles exact duplicates. Fuzzy duplicates — “John Smith” and “J. Smith” referring to the same person — are a different problem entirely. The recordlinkage library handles those cases, but that’s a topic of its own.

 [INTERNAL LINK: entity resolution / record linkage article if available]

Clean data doesn’t mean perfect data. It means data that’s consistent enough to learn from.

Step 2 — Data Integration: Merging Without Breaking Things

Real-world projects draw information from multiple sources. Information on sales figures from one database, demographics of customers from another, and behavior on the internet from yet another. Integration merges them, but inevitably creates new issues.

Schema Matching and Resolving Conflicts

Integration is not possible without first resolving schema conflicts. Different column names in different databases. Differences in data types. Mismatched units of measure.

import pandas as pd

# Standardize column names before merging
sales_df.rename(columns={'cust_id': 'customer_id'}, inplace=True)

# Match data types
sales_df['customer_id'] = sales_df['customer_id'].astype(int)
customers_df['customer_id'] = customers_df['customer_id'].astype(int)

# Merge
merged = pd.merge(sales_df, customers_df, on='customer_id', how='inner')

The type of join is crucial. The inner join will retain only those records for which there is a corresponding record in both tables. A left join will retain all rows from the left table, regardless of whether there is a corresponding entry in the right table. If you are not sure of the degree of overlap in your two sources, go for a left join first and analyze your NaN entries.

Handling Redundancy After Integration

Once you have integrated data, you will find that some columns convey almost the same information. Columns with a correlation coefficient greater than 0.95 should be considered redundant.

python
corr_matrix = merged_df.corr().abs()
upper_triangle = corr_matrix.where(
    pd.np.triu(pd.np.ones(corr_matrix.shape), k=1).astype(bool)
)
to_drop = [col for col in upper_triangle.columns if any(upper_triangle[col] > 0.95)]
merged_df.drop(columns=to_drop, inplace=True)

Integration done well creates a richer dataset. Integration done carelessly creates a noisier one.

Step 3 — Data Transformation: Getting Data Into Model-Ready Shape

Raw numbers cannot be used by many algorithms. A salary feature between 20,000 and 500,000 will overshadow another feature between 1 and 5 unless scaled appropriately. Feature scaling addresses that issue and does much more.

Feature Scaling — Normalization (Min-Max scaling) vs. Standardization (Z-Score scaling): Which One and When?

That is the most commonly searched question regarding this topic; it warrants a clear-cut answer rather than “it depends.”

Normalize your features (use Min-Max scaling) when:

• You use distance-based algorithms: KNN, SVM, neural networks
• There are no severe outliers
• You need to normalize your data to a specific range: 0 to 1 for image pixels

Standardize your features (use Z-score scaling) when:

• You apply linear models, logistic regression, or PCA
• Your data approximates a normal distribution
• There are outliers present which have not yet been eliminated: StandardScaler is better at handling them than MinMaxScaler

Use RobustScaler when there are significant outliers that you cannot eliminate.

python
from sklearn.preprocessing import MinMaxScaler, StandardScaler, RobustScaler

# Min-Max
minmax = MinMaxScaler()
X_train_mm = minmax.fit_transform(X_train)
X_test_mm = minmax.transform(X_test)

# Z-Score
standard = StandardScaler()
X_train_std = standard.fit_transform(X_train)
X_test_std = standard.transform(X_test)

# Robust
robust = RobustScaler()
X_train_rob = robust.fit_transform(X_train)
X_test_rob = robust.transform(X_test)

Encoding Categorical Variables

Models require numbers. You have to encode categories into numbers, but how you do this alters the way your model perceives their relationship.

One-Hot Encoding – when you have nominal categories without any particular order like cities or products or colors, you assign one binary variable for each category.

Ordinal Encoding – use for categorical features that have an order like low/medium/high or small/medium/large.

Avoid label encoding on nominal variables. When you encode “Paris,” “London,” “Tokyo” as 0, 1, 2 respectively, you are implicitly telling your model that “Tokyo” is double that of “London.”

python
from sklearn.preprocessing import OneHotEncoder, OrdinalEncoder

# One-Hot for nominal
ohe = OneHotEncoder(sparse_output=False, handle_unknown='ignore')
X_train_ohe = ohe.fit_transform(X_train[['city', 'product_type']])
X_test_ohe = ohe.transform(X_test[['city', 'product_type']])

# Ordinal for ordered
oe = OrdinalEncoder(categories=[['low', 'medium', 'high']])
X_train_ord = oe.fit_transform(X_train[['priority']])
X_test_ord = oe.transform(X_test[['priority']])

Feature Engineering — Creating Better Inputs

The answer may not even be found in your data set; it could be an outcome of the data you have. 

The column purchase_date will not help much when considered alone. Extract the day_of_week and month from this column, then you capture the seasonal aspects and weekend shopping tendencies. Both aspects are helpful for a model to predict the outcomes. From revenue and cost, derive profit_margin = (revenue – cost) / revenue. That ratio might carry more predictive signal than either column on its own.

python
df['day_of_week'] = pd.to_datetime(df['purchase_date']).dt.dayofweek
df['month'] = pd.to_datetime(df['purchase_date']).dt.month
df['profit_margin'] = (df['revenue'] - df['cost']) / df['revenue']

Surprisingly, the process of feature creation through domain knowledge plays a more important role than scaling techniques. The algorithm cannot detect the presence of any pattern that you have not provided as part of the features.

Data transformation is not aimed at cleaning your dataset; rather, it is concerned with exposing patterns for detection by an algorithm that does not understand context as humans do.

Step 4 — Data Reduction: Less Is More

But more features aren’t always better. Unrelated features are distracting. Similar features slow down the learning process. And both are detrimental to our algorithm. Dimensionality reduction gets rid of both kinds.

Feature Selection — Keeping What Matters

Three main strategies, each with its own strength.

Filter methods rely on the statistical dependence between each feature and the dependent variable. Chi-squared test is used for classification, while Pearson correlation for regression problems. It’s fast and independent of models.

Wrapper methods like RFE (Recursive Feature Elimination). It works similarly to filter methods except that at every round, it trains the model and eliminates the least relevant feature based on the chosen criterion.

The embedded method delegates feature selection to our model. Some models do just that. Examples include feature importance from random forests and LASSO regressions.

python
from sklearn.feature_selection import SelectKBest, chi2, RFE
from sklearn.linear_model import LogisticRegression

# Filter method
selector = SelectKBest(chi2, k=10)
X_train_filtered = selector.fit_transform(X_train, y_train)
X_test_filtered = selector.transform(X_test)

# Wrapper method (RFE)
estimator = LogisticRegression(max_iter=1000)
rfe = RFE(estimator, n_features_to_select=10)
X_train_rfe = rfe.fit_transform(X_train, y_train)
X_test_rfe = rfe.transform(X_test)

Dimensionality Reduction with PCA

PCA reduces your features to a smaller number of uncorrelated features based on their variance. It saves time during training and reduces the amount of correlated noise in your data.

However, do not apply PCA if you need a comprehensible model. Once your features are reduced to mathematical combinations of their values (components), you cannot say “Feature X caused this prediction.” The lack of interpretability is unacceptable in regulated industries such as banking and medicine.

python
from sklearn.decomposition import PCA

# Retain 95% of variance automatically
pca = PCA(n_components=0.95)
X_train_pca = pca.fit_transform(X_train_scaled)
X_test_pca = pca.transform(X_test_scaled)

print(f"Reduced from {X_train_scaled.shape[1]} to {X_train_pca.shape[1]} components")

Reduction makes your pipeline faster and often more accurate — but only when you’re cutting noise, not signal.

The One Rule All Three Steps Depend On — Fit on Train, Transform on Test

It has happened to me many times to see in data science tutorials how they fit their preprocessing pipeline over the entire dataset and then praise themselves for having a proper workflow. However, at that point, you have already violated the process of evaluation before training.

This is what you shouldn’t do:

python
# WRONG -- leaks test statistics into training
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split

scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)          # Fitting on ALL data
X_train, X_test = train_test_split(X_scaled, test_size=0.2)
And the right way:
python
# RIGHT -- test set stays truly unseen
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42
)

scaler = StandardScaler()
X_train_scaled = scaler.fit_transform(X_train)  # Fit ONLY on train
X_test_scaled = scaler.transform(X_test)         # Transform, never re-fit

The cleanest way would be using sklearn’s Pipeline, which forces this distinction through its design – transformers are fit only on the training set, and they are used on the test set automatically.

python
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.impute import SimpleImputer
from sklearn.linear_model import LogisticRegression

pipeline = Pipeline(steps=[
    ('imputer', SimpleImputer(strategy='median')),
    ('scaler', StandardScaler()),
    ('model', LogisticRegression(max_iter=1000))
])

pipeline.fit(X_train, y_train)
print(pipeline.score(X_test, y_test))

Every model going to production should be built this way. Not because it’s cleaner code — though it is — but because it makes data leakage structurally impossible.

Tools for Data Preprocessing in 2026

Tool	Best For	Scale	Skill Level
Pandas	Tabular manipulation, cleaning	Small–Medium	Beginner
NumPy	Numerical array operations	Small–Medium	Beginner
scikit-learn	ML preprocessing + pipelines	Small–Medium	Intermediate
Dask	Datasets that exceed available RAM	Large	Intermediate
Great Expectations	Automated data validation	Any	Intermediate
AWS Glue	Cloud-scale ETL pipelines	Enterprise	Advanced
Azure Data Factory	Cloud data integration	Enterprise	Advanced

Dask is very similar to Pandas, so it is easy to learn. Dask operates on chunks, which means that a 50 GB CSV file that could not be handled locally by Pandas will work fine.

Great Expectations is the package that is rarely mentioned during data preprocessing tutorials. You can create rules for data validation — such as “this column should never be empty,” “the values in this feature should always be within 0 and 1” — and check whether they hold prior to training any models. If something goes wrong with your preprocessing, you will get an alert immediately rather than three days after the predictions start to drift.

In reality, most practicing data scientists begin with Pandas and scikit-learn, transition to Dask once they face scalability issues, and finally integrate Great Expectations into their pipeline.

Data Preprocessing Best Practices That Actually Matter

Preserve your raw data. Once you apply transformations to your data and replace the original values, you will lose them forever. Apply version control techniques to your data in the same manner as you apply version control to code.

Run EDA before Preprocessing. Examine the distribution, rate of missing values, and presence of outliers in your dataset before making any decisions regarding how to process the data. Decisions made without context must be corrected or reversed later.

Use pipelines, not notebooks, for production. Exploration is done in notebooks but anything running on a schedule or used to train machine learning models needs to be contained within a pipeline.

Document every decision. Why did you choose median over mean? Why did you drop the column instead of imputing? Record all decisions regardless of choice because it will be important for the reasoning behind them in future collaborations.

Validate output before training.. Verify that your output dataset is correctly formed, contains no unexpected null values, and the range of your data is reasonable.

Treat this as iterative. Your initial decisions concerning preprocessing may change. Model performance can guide you back.

Common Data Preprocessing Mistakes (And How to Avoid Them)

Mistake	What Goes Wrong	Fix
Fitting scalers before splitting	Test statistics contaminate training	Split first; use Pipeline
Imputing with full-dataset statistics	Same leakage, different route	Fit imputers only on X_train
Label encoding nominal categories	Model treats category order as real	Use OneHotEncoder for nominal data
Dropping columns too aggressively	Useful signal discarded permanently	Use correlation threshold + feature importance before dropping
Not applying preprocessing to inference data	Production model receives raw, unscaled inputs	Save and reuse the fitted Pipeline object via joblib
Re-fitting transformers per batch	Transformer statistics drift in production	Fit once, serialize, reuse

Frequently Asked Questions