Random Forest Algorithm: How It Works, When to Use It

Any data scientist worth his or her salt pulls out the XGBoost as soon as there’s some competition involved. This is always a bad idea.

The algorithm known as Random Forest was developed by Leo Breiman in 2001 and trademarked along with Adele Cutler. It remains the first thing you should test with any tabular data challenge, not because it’s going to be the most precise one. Because it is a trustworthy baseline: decent performance off the bat, almost zero preparation needed, and an automatic validation system that most people aren’t even aware of.

This piece is not about how Random Forest operates. Instead, it is about something way more important than that.

What Is Random Forest?

Random Forest is a machine learning method that uses supervised training of numerous decision trees on randomly selected sub-samples of your data in order to make ensemble predictions by either voting for classification problems or averaging for regression problems.

The idea suggested by Leo Breiman back in 2001 in his paper published in Machine Learning is that an ensemble of weaker but diverse learners will outperform the strongest learner on average. Each of the trees used in a forest works with a different subset of the data and gives its prediction; then the forest makes an ensemble prediction that will be better than any of the predictions made by separate trees.

How Random Forest Works: Step by Step

Three mechanics make Random Forest different from a single decision tree. Most articles list them. This one explains why each one actually matters.

Step 1: Bootstrap Sampling

Each time before the tree is trained, the algorithm randomly samples your training data. And here comes the important bit: the sampling is done with replacement, meaning that the same line can be in the sample more than once, while some lines will be completely missed.

On average, each tree uses only around 63.2% of all unique lines of your dataset to be trained. The rest ~36.8% remain unused and have a special name: out-of-bag samples. We will talk about their importance later.

It’s called “bagging,” or bootstrap aggregating. It was invented by Breiman in 1996, 5 years before Random Forest. Each tree receives its own subset of the data, meaning that each tree will make its own mistakes.

And averaging mistakes that do not correlate with each other increases the accuracy. This is the entire process, no sorcery, but simple math.

Step 2: Feature Bagging (The Part Most Explanations Gloss Over)

The Bootstrap alone is not enough to understand how good Random Forest is doing. There’s another component called the feature bagging or the random subspace method.

At each split in each individual tree, the algorithm does not take into account all features in your dataset. Instead, it selects a random subset of features, usually equal to the square root of the total number of features when we’re doing classification.

Here’s a catch. If we have one feature which is a lot better than others, like income in a credit risk model, without feature randomness, every tree will use income as the splitting criteria and create trees which are basically the same. Identical trees mean identical errors. We’ll have a forest which is nothing but the same decision tree in different guises.

Feature bagging creates diversity. Different trees. Errors become uncorrelated.

Step 3: Making Predictions

For classification: each tree votes for a class, and the majority class wins.

For regression: each tree gives a number, and the average of all numbers is returned.

No weights. No corrections. Just a vote.

Out-of-Bag Evaluation: A Free Validation Tool

Most users reserve 20% of the dataset for validation before training commences. This is reasonable. However, with Random Forest, you can do without it.

Do you recall the roughly 36.8% of the rows which each individual tree doesn’t see while it is being trained? Well, you can validate each particular tree on data that it hasn’t seen. And when you do that for all the trees, you will have an OOB error estimate.

In scikit-learn, this can be achieved by using one parameter only:

python
from sklearn.ensemble import RandomForestClassifier

rf = RandomForestClassifier(n_estimators=200, oob_score=True, random_state=42)
rf.fit(X_train, y_train)

print(f"OOB Score: {rf.oob_score_:.4f}")

OOB statistic is quite beneficial to use when you have a smaller sample size where leaving out 20% can adversely impact the quality of learning. In reality, however, OOB and CV estimates come pretty close, but OOB is certainly quicker and more efficient than CV.

Implementing Random Forest in Python (scikit-learn)

Classification Example

The code below uses the breast cancer dataset from sklearn: clean, real, no download needed, and more informative than the Titanic example you’ve probably already seen ten times.

python
from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import train_test_split
from sklearn.metrics import accuracy_score, classification_report

# Load data
data = load_breast_cancer()
X, y = data.data, data.target

# Split -- stratify keeps class ratios consistent between train and test
X_train, X_test, y_train, y_test = train_test_split(
    X, y, test_size=0.2, random_state=42, stratify=y
)

# Train with OOB scoring enabled
rf = RandomForestClassifier(
    n_estimators=200,
    oob_score=True,
    random_state=42,
    n_jobs=-1   # trains across all CPU cores in parallel
)
rf.fit(X_train, y_train)

# Evaluate
y_pred = rf.predict(X_test)
print(f"Test Accuracy:  {accuracy_score(y_test, y_pred):.4f}")
print(f"OOB Score:      {rf.oob_score_:.4f}")
print("\nClassification Report:\n", classification_report(y_test, y_pred,
      target_names=data.target_names))

Two parameters worth explaining before you move on.

stratify=y keeps the class ratio consistent between train and test. Skip it on imbalanced data and you might accidentally push almost all the minority class into the test set.

n_jobs=-1 tells scikit-learn to use every available CPU core. Random Forest trees train independently of each other, so they parallelize cleanly. On an 8-core machine, training 200 trees takes roughly the same time as training 25.

Checking Feature Importance

After training, you can ask the model which features drove its decisions.

python
import pandas as pd
import matplotlib.pyplot as plt

# Gini-based feature importances
importances = pd.Series(
    rf.feature_importances_,
    index=data.feature_names
).sort_values(ascending=False)

importances[:10].plot(kind='barh', figsize=(8, 5))
plt.title("Top 10 Feature Importances (Mean Decrease in Impurity)")
plt.tight_layout()
plt.show()

One honest caveat: Gini-based importance can overstate high-cardinality numerical features. If you need more reliable rankings for a report or a regulated environment, use permutation importance instead:

python
from sklearn.inspection import permutation_importance

perm = permutation_importance(rf, X_test, y_test, n_repeats=10, random_state=42)
perm_importances = pd.Series(
    perm.importances_mean,
    index=data.feature_names
).sort_values(ascending=False)

Permutation importance measures how much accuracy drops when each feature’s values are shuffled. It’s a more direct, less biased measure of actual predictive contribution.

Hyperparameter Tuning: The Three That Actually Matter

All tuning guides include all these parameters from the documentation. The answer is simple – three parameters dictate most of the improvements.

n_estimators tune the number of trees. Additional trees decrease variance but provide diminishing returns beyond a certain point. Usually, after having around 200–500 trees, additional trees do not change the accuracy while increasing the training time linearly. Start with 200, and increase only if the OOB score is still improving.

max_depth is the parameter responsible for the depth of each tree. When not limited (‘max_depth=None’, the default setting), the tree grows until all leaf nodes become pure – that is, it starts overfitting on noisy data. A good initial value is a range between 10–20. To be honest, it is the most overlooked parameter by most newbies.

max_features dictates how much randomness is added to the model. Its default ‘sqrt’ works perfectly in most situations. The lower value increases tree diversity but decreases its individual quality. Setting it to 1.0 removes the tree’s randomness and transforms your random forest into bagged trees.

python
from sklearn.model_selection import RandomizedSearchCV
from scipy.stats import randint

param_dist = {
    'n_estimators': randint(100, 500),
    'max_depth': [None, 10, 15, 20, 30],
    'max_features': ['sqrt', 'log2', 0.5],
    'min_samples_leaf': randint(1, 5)
}

search = RandomizedSearchCV(
    RandomForestClassifier(random_state=42, n_jobs=-1),
    param_distributions=param_dist,
    n_iter=20,
    cv=5,
    scoring='f1_weighted',  # use f1, not accuracy, on imbalanced data
    random_state=42
)
search.fit(X_train, y_train)
print("Best params:", search.best_params_)

Handling Imbalanced Data with Random Forest

Most training data sets are balanced and noise-free. Real-world applications are not.

Fraud detection, disease diagnosis, equipment malfunctioning. Almost always, whatever is important to predict is the minority class. A machine learning algorithm that labels everything as non-fraudulent can achieve 99% accuracy on a data set where only 1% of records are fraudulent. Such an accuracy measure is completely useless.

Random Forest offers you two possibilities in such a scenario.

Option 1: class_weight=’balanced’

python
rf_balanced = RandomForestClassifier(
    n_estimators=200,
    class_weight='balanced',  # weights classes inversely to their frequency
    oob_score=True,
    random_state=42
)
rf_balanced.fit(X_train, y_train)

This adjusts each tree’s split criterion to penalize mistakes on the minority class more heavily. Simple, fast, often enough.

Option 2: Oversample with SMOTE before training

In contrast, SMOTE creates synthetic instances of the minority class and not just assigns different weights. It usually works better in highly imbalanced datasets, where ratios can go down to 1:50 or even lower.

python
from imblearn.over_sampling import SMOTE

smote = SMOTE(random_state=42)
X_resampled, y_resampled = smote.fit_resample(X_train, y_train)

rf.fit(X_resampled, y_resampled)

In case you have a dataset with imbalanced classes, never use accuracy to measure your model performance, but precision, recall, and F1 score instead. These metrics are provided to you by the classification report – you just need to check the row corresponding to the minority class.

How to Interpret Random Forest Predictions with SHAP

There is one thing that you will not find out in most Random Forest tutorial guides – importance of features shows which variables matter globally when making any prediction. The importance of features does not show the reason for a certain prediction on a particular object.

It makes sense in practice because a credit officer does not need to be told “income is the most important feature” – he needs to be shown why this applicant was denied.

SHAP (SHapley Additive explanations), suggested by Lundberg and Lee, can help here. It calculates the contribution of each feature for every particular prediction using the game theory. There is an efficient implementation of SHAP for trees called TreeExplainer.

python
import shap

# Create explainer
explainer = shap.TreeExplainer(rf)

# Calculate SHAP values for test set
shap_values = explainer.shap_values(X_test)

# Summary plot: global importance + direction of effect
shap.summary_plot(shap_values[1], X_test, feature_names=data.feature_names)

# Explain a single prediction
shap.waterfall_plot(
    shap.Explanation(
        values=shap_values[1][0],
        base_values=explainer.expected_value[1],
        data=X_test[0],
        feature_names=data.feature_names
    )
)

The Summary Plot indicates the direction of Gini importance in terms of the effect of each variable on pushing the predictions into either class 1 or class 0 and how this direction varies over the values of the variables. The Waterfall Plot explains one prediction only, variable after variable.

In the regulated industries of banking, insurance, and healthcare, SHAP values are becoming increasingly the thing that the compliance team is asking for.

Random Forest vs. XGBoost: When to Use Each

In my opinion, this way of looking at it is completely reversed.

Typically, articles refer to XGBoost as the “better” method, whereas Random Forest is presented as the “simpler” method that people use when they are still learning how to work with data. This way of thinking is incorrect from a professional standpoint.

When Random Forest Wins

Your data is noisy. Random Forest’s averaging is more forgiving of noisy features than XGBoost, which can fit noise aggressively when boosting is pushed too far.

You need fast results without much tuning. A default Random Forest with 200 trees usually gives you 80–90% of achievable performance on a new dataset with no hyperparameter work. XGBoost needs more careful configuration to reach its ceiling.

Your dataset is large. Trees train independently, so the work spreads across all your CPU cores with n_jobs=-1. XGBoost’s sequential boosting doesn’t parallelize across cores as cleanly, though it has GPU support.

You want free validation. OOB scoring is a genuine advantage on small-to-medium datasets where you can’t afford to lose 20% of your data to a holdout set.

When XGBoost or LightGBM Wins

You will be fighting in a competition. It has been proved again and again that XGBoost and LightGBM have a higher upper bound for accuracy on structured datasets; there is no way around it.

Your dataset is small. Boosting’s incremental learning through error correction will perform better for smaller amounts of data. With less data, training hundreds of different trees becomes difficult; the emphasis on hard samples becomes important.

You want to make faster predictions in production. One single XGBoost model with shallow trees usually will make predictions faster than one Random Forest model with 500 trees trained very deeply.

You can spend extra time tuning. XGBoost has more parameters to tune, more interactions between them, and a higher upper bound for performance. But only if you do it right.

Random Forest Advantages and Limitations

Advantages

No feature scaling needed. Decision trees perform splits based on threshold value, not distance/magnitude, therefore normalizing your dataset makes no difference for the splitting process. Forget about it.

Built-in feature importance. All trained models give you feature_importances with no extra work. It’s not a perfectly accurate metric, but it’s an easy and quick way to see which features are utilized by your model.

Naturally parallel. Decision trees are independent of each other when being trained, thus utilizing all CPU cores at once.

Handles missing values natively. Starting from scikit-learn 1.4, RandomForestClassifier supports NaNs during training without any preprocessing. At each split, the tree grower chooses a direction (left or right) which will have the smallest impurity for these missing values.

Hard to break. It may not win every single competition, but it’s really hard to screw up Random Forest. This property is underappreciated in production where robustness is valued higher than 0.3% of accuracy boost.

Limitations

Memory in production. A forest of 500 deep trees takes quite a bit of memory. On edge devices, serverless functions, or low-RAM containers, this is a meaningful overhead. A shallower XGBoost model with just one tree is a better choice there.

Prediction latency at scale. Every prediction needs to traverse all the trees. Having 1,000 trees and deep splits means having an overhead. This starts mattering if you need to deliver millions of predictions per second and care about latency.

Can’t extrapolate. Tree-based models can’t predict outside the bounds of input values they were trained on. So if you have a $100,000-$1,000,000 house price range in your training data, your model will never know what to predict for $3 million houses. You’ll get clipping. It’s a hidden pitfall for regressions.

Interpretability at the individual level. One decision tree is interpretable. The forest of 500 trees isn’t. Use SHAP for prediction explanations instead of a bar chart of feature importance.

Real-World Applications of Random Forest

Finance. In terms of credit scoring and fraud detection, the Random Forest algorithm has certainly been used for the longest time. Resilience to overfitting on dirty transaction data and an easy-to-implement training process are what makes it a realistic starting point. Fraud detection systems use the RF algorithm to make initial filters even if the final classification is done through gradient boosting.

Healthcare. Random Forests have been extensively applied to EHR (electronic health records) data for estimating risk scores, readmissions, and drug response predictions. In Breiman’s (2001) paper, gene expression classification was one of the application domains of Random Forest, and the scope of applications has greatly broadened since then.

Environmental science. The research by Mazuruse et al., conducted in 2026, utilized a Random Forest classifier to make air pollution forecasts for major cities in Africa, such as Lagos and Nairobi, based on the data obtained in more data-driven cities, which were then transferred to less data-driven cities. Nonlinearity in environmental data was the reason for the selection.

E-commerce and recommendations. Random Forest models are used by recommendation engines as ranking models: when presented with a user and a list of possible products, it predicts the probability of clicking or purchasing on each of the products. Feature importance helps to understand what actions of users cause conversion, which is valuable knowledge for product managers.

Hydrology and climate. Applications include flood hazards mapping, risk assessment for wildfires, and prediction of crop yields. The dominance of tree-based ensembles in applied environmental machine learning literature is due to their ability to deal with nonlinear, multivariate, and spatially heterogeneous data without assuming any distributional properties.

Frequently Asked Questions

Closing Thought

25 years after Breiman’s work, those practitioners who will benefit most from the Random Forest are not the ones that see it as an intermediate step on their way to XGBoost; they’re the ones that see it as the gold standard others need to surpass.

First run a baseline RF. Every time. Analyze its failures, understand why, and only then consider the necessity of applying some more complex model that takes more effort to tune.

An algorithm that needs the most tuning is not necessarily the best solution to your task. This is pretty much the core of the concept.