Decision Tree in Machine Learning

The decision tree algorithm can perform perfectly on the training data set and fail the very first time it meets a new example. This is not a bug. Rather, this is the most common mistake people make with this machine learning algorithm and virtually no one points it out before going into details.

This article is doing the opposite intentionally. First, we are showing how things go wrong and why it happens, before getting into technical details and definitions.

What Is a Decision Tree?

The decision tree is a type of supervised learning algorithm which generates predictions by posing yes or no questions pertaining to the data in order to arrive at an answer.

Think of a flow chart. Here, the root node is at the very top, and this is the most important question that is asked regarding the data. Thereafter, there are other internal nodes which ask more questions depending upon the result of the previous question. Thereafter, there are leaf nodes which provide the result – a category such as “spam” or “non-spam” or a numerical prediction. The lines connecting these are called branches, and each one represents the outcome of a test, yes or no, above or below a threshold.

That’s the whole idea, honestly. It’s a flowchart that a computer builds for you instead of one you draw by hand.

How a Decision Tree Actually Makes a Decision

Let’s try something that matters for once instead of the usual theoretical example. A regional bank needs to mark the applications likely to become defaulting, basing on three features – the income of an applicant, his debt-to-income ratio, and whether he has made a late payment within the last year.

The tree does not consider all three features at once. First, it asks one question, let us say, “Has the applicant made a late payment within the last year?” since only one question manages to distinguish defaulters from non-defaulters in the training dataset better than any other question. The answer is yes – and the tree goes on to ask another question concerning the debt-to-income ratio. The answer is no – and the application gets accepted without further questions.

Here comes the unexpected thing. The bank’s data scientist constructing the tree discovered that income, which was considered the most influential variable, influenced the process very little. Late payments and debt ratio carried out virtually all the action. This is a common occurrence in credit risk modeling. Usually, payment history gives more accurate information about future payment behavior than income since income does not provide any information about management of existing financial resources.

The tree will continue splitting the data in such a manner step by step until the data becomes “pure” – meaning that all the applicants in the data set have the same outcomes. When purity is reached (or when a decision is made to stop) then the branch reaches its end and forms a leaf with a prediction.

Information Gain vs. Gini Index — Which One Actually Matters?

But what determines which question gets asked first? The tree needs a criterion for determining which question produces the best separation in the data. There are two widely used criteria for that purpose, and tutorials always cover both without explaining which one you should use.

Entropy and Information Gain measure uncertainty. The entropy of a set with equal numbers of outputs is very high because you have no idea what you are dealing with. The entropy of a set where all members share the same output is zero. Information gain simply is the reduction in entropy caused by asking a particular question.

Gini Index measures something very similar to this: how often you would have classified a randomly selected member incorrectly, assuming that the classification was made by considering the proportion of members in the group. A perfectly pure group gets a score of 0. A 50/50 group gets the maximum possible score of 0.5.

In reality, they end up making similar choices. The truth that will not be told in many papers: it doesn’t matter which one you select, not to the extent that all the hype about “selecting the right splitting criterion” makes it sound like it does. The Gini Index is just a bit quicker because it doesn’t need to use the logarithm function the way entropy does. This is why criterion=’gini’ is the default choice in scikit-learn. Unless you have a compelling reason to favor entropy (which some people claim treats mixed groups harsher), Gini Index is the smart default choice.

Building One in Python (A Working Example)

Reading about decision trees only gets you so far. Here’s one built and trained in under fifteen lines, using scikit-learn’s DecisionTreeClassifier.

from sklearn.tree import DecisionTreeClassifier

from sklearn.datasets import load_wine

from sklearn.model_selection import train_test_split

# Load a real dataset — wine chemistry data, predicting wine class

X, y = load_wine(return_X_y=True)

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

# Build and train the tree

clf = DecisionTreeClassifier(max_depth=4, random_state=42)

clf.fit(X_train, y_train)

# Check accuracy on data it's never seen

print(f"Test accuracy: {clf.score(X_test, y_test):.2f}")

# Which features actually mattered?

for name, importance in zip(load_wine().feature_names, clf.feature_importances_):

    if importance > 0.05:

        print(f"{name}: {importance:.2f}")

Running this on the wine dataset typically lands somewhere around 90-94% test accuracy, and feature_importances_ will usually show that a small handful of chemical properties, often color intensity and proline level, do most of the heavy lifting, while several other features barely register.

Notice the max_depth=4 parameter. Leave that out, and the tree will keep splitting until every leaf is perfectly pure, which sounds great until you remember the opening of this article.

Why Decision Trees Overfit (And How to Stop It)

Now for our first overfitting problem in action. In its native state, a decision tree algorithm will just keep splitting your training set into ever-smaller chunks until it reaches single data points. In this case, the algorithm isn’t learning a pattern; it’s just memorizing the training set.

This is overfitting, and it’s by far the most common issue when using this algorithm in practice. The tree will look great with the training set and humiliate you when working with any other dataset.

The solution is to limit the growth of the tree. The key variables that control this process are:

• max_depth restricts how many levels deep the tree is able to get. Small values will make the tree less complex, and more generalized.
• min_samples_split determines the minimal number of samples needed in order to allow a split. In cases where there are only two or three samples remaining, making a split at this level would not serve any purpose other than to fit noise.
• min_samples_leaf is used to specify the same restriction for leaves.

This process of deliberately limiting tree growth is called pruning. You can prune ahead of time by setting these parameters before training (pre-pruning), or you can let the tree grow fully and then trim back branches that don’t earn their keep afterward (post-pruning, often via scikit-learn’s ccp_alpha parameter). Either way, the goal is the same: a simpler tree that generalizes instead of memorizing. This is Occam’s Razor applied directly to machine learning: the simplest model that still explains the data usually wins.

A pruned tree that’s right 85% of the time on new data beats an unpruned one that’s right 100% of the time on old data and badly wrong on everything else.

Classification Trees vs. Regression Trees

Decision trees perform two types of prediction, and the process is almost identical for both of them.

Classification trees the task is to predict a class. Is this going to be a spam email? Is this loan going to be a defaulter? The leaf node provides the output as a class label.

Regression trees predict a numeric value. What would be the selling price of this house? How many units will be sold by this store next month? In this case, the leaf node does not give out a class label but a numeric value such as average.

This average is usually the mean of the training data that ends up in this particular leaf node.

Common Decision Tree Algorithms — ID3, C4.5, and CART

Algorithm	Creator	Splitting Metric	Still Used Today?
ID3	Ross Quinlan, 1986	Information Gain (entropy)	Mostly historical — foundational, rarely used directly
C4.5	Ross Quinlan, 1993	Gain Ratio	Influential; descendants still appear in some tools
CART	Leo Breiman et al., 1984	Gini Index (classification) or variance (regression)	Yes — this is what scikit-learn implements
CHAID	Gordon Kass, 1980	Chi-squared test	Still used in market research and survey analysis

If you’re using scikit-learn today, you are currently using a CART-based implementation. The significance of ID3 and C4.5 lies mainly in knowing about the background of the field. In any case, Quinlan’s original paper on ID3, based on a learning approach developed by the psychologist Earl Hunt in the 1960s, is definitely worth browsing through.

When to Use a Decision Tree (And When Not To)

But what about using it anyway? Here is how to make an easy decision.

Apply a decision tree if:

• You need to explain your model’s reasoning to someone non-technical, like a regulator, a doctor, or a loan officer. Few models are as easy to walk through out loud.
• Your dataset has a mix of numerical and categorical features without heavy preprocessing.
• You’re working with a small-to-medium dataset where training speed matters more than squeezing out the last bit of accuracy.
• You want a fast first model to understand which features matter before committing to something heavier.

Skip it, or at least don’t stop there, when:

• Your data is noisy, with a lot of borderline cases. Single trees are unstable and can flip dramatically with small changes in training data.
• You need maximum predictive accuracy and don’t need to explain every decision. This is when Random Forest or gradient boosting methods like XGBoost typically outperform a single tree by averaging hundreds of them together.
• You’re working with very high-dimensional data, like text or images, where trees tend to struggle compared to other approaches.

The decision tree will usually not be the final model used in any serious production process. The decision tree will usually be the initial model and an excellent diagnosis tool even when you have moved beyond it, since the importance of the features generated is still applicable to the new ensemble model.

Advantages and Disadvantages

Advantages

• Interpretability. You can trace every prediction back through a specific, readable path of questions. Few algorithms offer this.
• Minimal preprocessing. Decision trees handle numerical and categorical features without scaling, and tolerate missing values reasonably well compared to distance-based models like KNN.
• Flexibility. The same core algorithm handles both classification and regression with almost no change in approach.

Disadvantages

• Overfitting risk. As covered above, an unconstrained tree will happily memorize noise.
• High variance. Small changes in training data can produce a meaningfully different tree. This instability is exactly why Random Forest exists: average enough slightly-different trees together and the instability mostly cancels out.
• Greedy construction. At each split, the algorithm picks the locally best question without considering whether a different choice now might lead to a better tree overall. It doesn’t backtrack.

Frequently Asked Questions

Closing

Look back to the default prediction example above. Rather than simply saying “the tree fits,” the data scientist who developed it did feature importances, found out that income was underperforming, pruned the tree, and only after did he trust the predictions. This additional effort, of testing out the knowledge the tree gained rather than simply how well it performed, is what turns a good decision tree into a dependable one.

Build the tree and then figure out what it learned.