PCA in Machine Learning

Most machine learning courses will tell you that performing principal component analysis is the first thing you should do when you have too many features.

They’re out of date. where autoencoders can fit on your laptop computer and UMAP only a pip install away, the trick is understanding when to use PCA, and when it’s actually sabotaging your models.

This tutorial is the guide you need, not a step-by-step course that ends with wine. We’re giving you everything — the theory, the code, the comparisons, and the top five ways PCA silently sabotages your projects.

What is PCA in Machine Learning?

PCA is a way of compressing a dataset with a bunch of features that are related to each other into a new dataset that has a smaller number of features, known as principal components, and with minimal loss of information.

PCA was initially conceived by Karl Pearson in 1901. Its theoretical foundations were later laid down by Harold Hotelling in 1933, many years before computers had even become fast enough to compute it! In the meantime, scikit-learn has done the same in just two lines!

PCA is an unsupervised approach. You don’t tell it what labels your data should have. What you give it is your dataset, along with the information about the variances of its features. The principal components are arranged in such a way that each successive one captures more variance than the preceding ones orthogonally.

That’s all there is to it.

How PCA Actually Works (Without the Math Wall)

Let’s imagine a 3-dimensional cluster that looks like a pancake floating somewhere in space. Two directions in that pancake cover most of the variations, while the third one is negligible. What PCA does is find the relevant directions automatically, without any prior knowledge about the nature of your features.

It is quite easy to imagine such a process with two, three, or even four dimensions. But try imagining it with 50 or even 5,000 of them.

Step 1 — Standardize Your Features

PCA is susceptible to changes in scale. When one variable is scaled in dollars while another is scaled in kilometers, the dollar-scaled variable would dominate the variance computation simply due to its larger scale. Ensure that you apply StandardScaler to all features before proceeding with PCA.

This is the most frequent cause for PCA failing. Without standardizing, the first principal component would merely indicate which variable has higher raw scores.

Step 2 — Build the Covariance Matrix

The covariance matrix is a squared matrix showing the relationship between each pair of variables. A positive covariance means that both variables increase or decrease together, while a negative covariance means that when one increases, the other decreases, and an independent relationship gives zero covariance.

For a set of data containing p variables, this will be a matrix of p × p.This is the input for the next process.

Step 3 — Find Eigenvectors and Eigenvalues

The eigenvector of the covariance matrix refers to a direction, while the eigenvalue indicates the amount of variance along that particular direction. If the eigenvalue is large, then there is a high amount of variance along the direction; otherwise, if the eigenvalue is small, then it indicates that the variance along that direction is negligible.

Arrange the eigenvectors based on their corresponding eigenvalues in descending order, and you will have prioritized your principal components based on usefulness. It’s really that simple!

To make things simpler for yourself, remember that scikit-learn uses the Singular Value Decomposition (SVD) of the data matrix rather than calculating the covariance matrix explicitly.

Step 4 — Choose How Many Components to Keep

The 95% variance rule is the practitioner’s default. The procedure requires plotting the cumulative variance accounted for and selecting the minimum number of components needed for the cumulative variance to exceed 95%, which, in MNIST, will range from around 150 to 160 pixels out of 784, depending on how one preprocesses the data. We have effectively reduced our number of variables by 80%, losing little.

The Kaiser criterion says all components with an eigenvalue above 1 are retained. Simple and clear, the rule, however, results in over-retention in small samples and under-retention in large samples.

A scree plot is simply the graphical depiction of the Kaiser rule. Eigenvalues are plotted in descending order, and the practitioner finds the “elbow,” the point beyond which further eigenvalues represent mostly noise. It’s a judgment call; there’s no exact calculation for finding the elbow.

In practice, use the 95% variance rule. Use the scree plot for understanding.

Step 5 — Project Your Data Onto the New Axes

Simply multiply your normalized values by the matrix of chosen eigenvectors. That’s all there is to it! You have reduced the number of columns, made them uncorrelated, and retained most of the original data.

Section takeaway: No discernible patterns are being found by PCA; it is simply ranking directions.

PCA in Python — A Working Example

In practice though, you don’t write any of this by hand. Here’s the working pipeline:

Python
import numpy as np
import pandas as pd
from sklearn.preprocessing import StandardScaler
from sklearn.decomposition import PCA
from sklearn.datasets import load_wine

# Load data
data = load_wine()
X, y = data.data, data.target

# Standardize -- do not skip this
scaler = StandardScaler()
X_scaled = scaler.fit_transform(X)

# Apply PCA
pca = PCA(n_components=0.95)  # keep 95% of variance
X_pca = pca.fit_transform(X_scaled)

print(f"Original features: {X.shape[1]}")
print(f"Components kept: {X_pca.shape[1]}")
print(f"Variance explained: {pca.explained_variance_ratio_.sum():.3f}")
print(f"Per component: {pca.explained_variance_ratio_}")

Apply this to the wine dataset, and usually, you will only get 8 features out of 13, capturing 95% of the variance. That is a minor improvement, but it is an honest one. PCA is not a miracle – in small, curated datasets, it hardly improves anything at all.

The n_components=0.95 hack is often overlooked in tutorials since n_components=2 is always hardcoded just to have a graph. In reality, however, the variance threshold should do the trick for you.

When to Use PCA — and When Not To

Honestly, this is the section that should come before any tutorial. Most articles skip it.

Use PCA When

You have many numerical features, and these are highly correlated. The PCA algorithm works best for cases where features do overlap in terms of information content, such as physical sensors, financial metrics, and gene expression profiles. PCA algorithms leverage correlations between features, and in their absence, very little is gained.

You need a fast and deterministic preprocessing phase prior to using a machine learning model. This makes PCA an efficient choice since it can be calculated quickly and deterministically.

You need a two-dimensional visualization of high-dimensional data. This is done through two principal components in just one plot.

Skip PCA When

The dependencies between variables are non-linear. PCA can only detect linear associations. If your input variables live on a manifold that is not flat — such as image or text embeddings generated by a neural network — use UMAP or t-SNE.

The input features are mostly categorical. One can do one-hot encoding for the categorical variables and apply PCA to them; it happens and generally does not provide any benefit because the variance structure of indicator variables cannot be straightforwardly translated into PCA.

You have to give some explanations to the regulator or some stakeholder who cannot interpret principal components intuitively. “The component consists of a weighted average of age, income, and tenure” won’t do. “The component is 0.3 × age + 0.4 × income − 0.2 × tenure” even less so.

Deep learning is already involved in your algorithm. Then you already have an autoencoder that finds its own representation and handles non-linearity without further ado.

Section takeaway: The honest framing is that PCA is a great tool for a narrowing set of problems, not a default move.

PCA vs t-SNE vs UMAP vs Autoencoders

To be fair, no single technique wins in every dimension. Here’s how the four most-used methods stack up:

Method	Linear?	Speed on 1M rows	Preserves structure	Best for
PCA	Yes	Very fast	Global	Linear data, fast preprocessing, regulated industries
t-SNE	No	Slow	Local only	Visualizing clusters in 2D — not for downstream modeling
UMAP	No	Fast	Global + local	Modern default for visualization and embedding compression
Autoencoders	No	Slow to train, fast to use	Learned, task-specific	Deep-learning pipelines, non-linear data, very large datasets

t-SNE dominated the late 2010s, largely because Laurens van der Maaten’s initial paper created gorgeous images of MNIST. Introduced in 2018 by Leland McInnes, UMAP is faster, more scalable, and retains more global information. In visualization in 2026, UMAP is generally the better default.

Autoencoders are the deep learning approach. They train a compressed feature set optimized for your particular dataset. The downside is computation time, sensitivity to hyperparameters, and a potentially brittle algorithm. The upside is that autoencoders can represent non-linear relationships impossible for PCA.

PCA excels in three categories: computational speed, deterministic output, and ease of interpretation. These are not trivial qualities.

PCA vs LDA vs K-Means

Strangely enough, people tend to confuse these three algorithms despite their very distinct purposes.

LDA (Linear Discriminant Analysis) is supervised learning. It finds discriminating directions using class labels, whereas PCA searches for directions with maximum variance. If class labels are available, then LDA will almost always outperform PCA when used for classification.

K-means is an unsupervised clustering algorithm that assigns points into clusters. Dimensionality reduction is not its goal, though it can be combined with PCA to reduce dimensions before clustering.

PCA is unsupervised dimensionality reduction. LDA is a supervised dimensionality reduction. K-means is unsupervised clustering.

Common Mistakes That Break PCA

That seems intuitive until it doesn’t. Here’s a list of five things that I see people doing wrong all the time:

1. Forgetting to scale. Mentioned above, but repeated since this is by far the biggest culprit for poor PCA performance. If your biggest-scale feature dictates PC1, you did not scale.
2. Running PCA before splitting train and test. This constitutes data leakage: the model “knows” what is in the test dataset, thus inflating its performance scores. Always: Splitting before scaling and fitting the scaler and PCA only on the train, followed by transformation on the test. Even if it seems like a trivial step.
3. Keeping too many components. There are cases where someone keeps 99% variance just “to be on the safe side.” But this defies the whole concept behind PCA: Compressing the number of features. Keeping 99% variance from 100 features might result in having 90 components – not much gain in compression at all.
4. Trying to interpret PC1 literally. “PC1 is the sum of age and income minus tenure.” This means absolutely nothing since PC1 doesn’t really represent anything.
5. Running PCA on one-hot encoded categories without consideration. The variability in the indicators’ data will be largely determined by their base frequency and not by any significant factor. PCA will be inappropriate if you have lots of categorical variables, consider multiple correspondence analysis instead.

Real Applications of PCA in 2026

However, PCA did not disappear with the advent of deep learning. On the contrary, PCA shifted towards particular areas where it was found to be advantageous.

Single-cell genomics is the biggest modern use case. The workflows based on Seurat and Scanpy rely on PCA as a preprocessing step before UMAP since single-cell RNA-seq data have 20,000+ genes, with most information being retained in the first 30-50 dimensions. While PCA can operate on millions of cells, UMAP cannot.

Eigenfaces is the classic computer-vision application,  which is the technique that Turk and Pentland developed for face recognition back in 1991. Although it is outdated for actual production applications, it remains an excellent illustrative example of how PCA works.

In quantitative finance, PCA is used to identify risk factors on yield curves and stock returns. The first three principal components on a yield curve usually represent level, slope, and curvature — and such a factorization technique has proven robust over many decades.

For anomaly detection, the application of PCA is unique. Reconstruct your data using the leading principal components and find the reconstruction error; any significant outlier is a suspect. It isn’t cutting-edge technology, but it is interpretable and fast.

This was pretty much the essence of PCA in 2026.

Advantages and Limitations

There’s indeed a good deal of information about both sides of PCA.

Advantages: PCA is quick and deterministic. PCA eliminates correlated variables. PCA compresses the data. PCA applies to all datasets regardless of size on all computers.

Limitations: PCA uses linear projections. The projection is difficult to understand. PCA is vulnerable to outliers and changes in scale. Probabilistic PCA uses normal distributions in order to have maximum likelihood interpretation — standard PCA doesn’t make any assumption about distributions at all. Finally, PCA may eliminate exactly the variance needed by your application, since PCA doesn’t care about applications at all.

The above drawback is important because PCA minimizes the wrong criterion.

PCA Variants Worth Knowing

To be honest, there are variants of PCA that compensate for particular shortcomings:

Kernel PCA copes with non-linear structure by transforming the data to a higher-dimensional space before running PCA. This method is slower and less easy to fine-tune, but its advantage is that it can identify structures that standard PCA would completely overlook.

Sparse PCA constrains the components to load only on a subset of the original variables. The resulting PCA models tend to be easier to interpret since they don’t have components such as PC1 = 0.1 times everything.

Incremental PCA processes the data in batches, making it possible to perform PCA on data that won’t fit in memory. This is the flavor of PCA to use when working with truly massive datasets.

The vast majority of people will never need to use any of these variants. What matters is knowing that they exist.

Frequently Asked Questions

Conclusion

In 2026, the trend that’s becoming clear is this: PCA isn’t going away, it’s becoming specific. By default, we’re visualizing using UMAP. By default, we’re compressing with autoencoders. PCA’s niche is the small, valuable space where speed, determinism, and interpretability matter together. This niche is genuine. It’s just smaller than before. Turn to PCA when you really want to do what it can do, not when you’ve forgotten how many other choices there are.