Naive Bayes Classifier

The Gmail spam filter has been keeping spam out of your inbox since 2004. The algorithm behind this filter is not a transformer. Not even a neural network. It is a probability formula that was written by an English minister back in the 18th century, long before there was any way to test it against empirical data.

This is Naive Bayes. And come 2026, when GPT-like algorithms are everywhere, it will continue to be in action in the organizations you interface with on a daily basis.

Not due to the laze of the engineers. But for some particular jobs, the algorithm wins hands down from any other classification algorithm in terms of efficiency, accuracy with smaller data sets, and especially the ability to do something that transformers cannot ever do – explain their decision in a simple way.

In this guide, you will learn about all of that – the math behind it, the novel application example, all three types of Naive Bayes algorithms with a decision-making model, sklearn python code implementation, and, most important of all, the answer to the one question that no Naive Bayes guide ever asked before – why does the wrong assumption lead to right predictions?

What Are Naive Bayes?

Naïve Bayes is a machine learning classification method based on probability, which evaluates the probability of the most probable class for the given dataset, taking into account prior probability and observation of the likelihood of features belonging to a particular class.

English Translation: It takes into account the amount of knowledge one has about the categories being considered and then looks at the attributes of the particular object and calculates probabilities to choose the most probable class.

Bayes, in this case, refers to Reverend Thomas Bayes, an 18th-century thinker, whose famous theorem, Bayes’ Theorem, was published after his death by his friend Richard Price in 1763, in the Royal Society’s Philosophical Transactions.

Price not only edited the work but might have contributed significantly to it, but wasn’t credited for his contribution.

What does ‘naïve’ mean? Naivety stems from the assumption that every attribute in the particular dataset is completely independent of the others. This is rarely the case in real-world situations, but it does not matter because it still works. We will return to this point later.

The Math Behind It (Bayes’ Theorem Explained Simply)

You don’t need a statistics degree for this. But you do need to understand three terms before the rest of the article makes sense.

What Bayes’ Theorem Actually Says

Here is the formula:

P(y|X) = P(X|y) × P(y) / P(X)

Expressed in language: “The probability that the answer is y, taking into account that I’ve noticed features X, equals the probability of observing X in cases of class y, multiplied by the base probability of y, divided by the total probability of X.”

Let us make it specific. You are a doctor. A person comes to see you and is suffering from a fever and a cough. You need to know the probability of their having the flu.

• P(y) — prior probability. What is the base rate of flu among patients coming to your practice this week? 20%.
• P(X|y) — likelihood. In what proportion do flu patients suffer from fever AND cough? 85%.
• P(X) — evidence. In what proportion do people come to see you suffering from a fever and cough? 30%.

So: P(flu | fever + cough) = 0.85 × 0.20 / 0.30 = 0.567

A 57% chance of flu. Not certain — but enough to start treatment while you wait for the test result.

This is Bayes in action: you start with a belief (20% base rate), observe evidence (fever + cough), and update your belief (57%). Each new piece of evidence shifts the probability. That updating process is the whole point.

The Naive Assumption (And Why It’s Brilliantly Wrong)

Here’s where the “naive” part comes in. Once the feature set contains several attributes, calculating P(X | y) for all combinations of features becomes computationally very costly. In a text classifier with a vocabulary size of 50,000 words, you will have to calculate joint probabilities of every possible combination of two words, three words, and so on.

The naïve Bayes classifier avoids this completely by making an assumption of independence of attributes. So instead of calculating P(fever AND cough | flu), it computes P(fever | flu) * P(cough | flu).

Formally: P(x₁, x₂, …, xₙ | y) = P(x₁ | y) × P(x₂ | y) × … × P(xₙ | y)

However, cough and fever don’t occur independently — they both originate from the same underlying immune system reaction. This is an incorrect assumption. However, we know that this is an incorrect assumption, and we make it nevertheless.

Why do we do that? It’s because the classifier is not concerned about getting the exact probability right — it is concerned with ordering the classes properly, ensuring that the answer gets a better score than all other alternatives. In this case, the wrong yet useful assumption of independence works surprisingly well, as its mistakes impact all classes in roughly the same way.

I believe this is the most undervalued idea in intro machine learning: usefulness often beats correctness. Naive Bayes makes mistakes in the numbers but gives the correct answer reliably enough to go into production.

The Full Classifier Formula

Strip away the denominator (it’s the same for every class, so it doesn’t affect which class wins) and you get:

Interpretation: “Select the class y which gives us the maximum value of the prior probability multiplied by the product of individual feature probabilities.”

This is the entire algorithm. Only one equation. Just three lines of Python. Let’s get into it.

Worked Example — Spam Email Classification

Every other tutorial uses a golf weather dataset. It’s fine for illustrating the math, but it tells you nothing about how Naive Bayes is actually used. So we’re going to build a tiny spam filter instead.

Setting Up the Problem

Say we have 5 training emails, and we’re tracking whether three words appear: “free,” “meeting,” and “invoice.”

Email	“free”	“meeting”	“invoice”	Class
1	Yes	No	No	Spam
2	Yes	No	Yes	Spam
3	No	Yes	No	Not Spam
4	No	Yes	Yes	Not Spam
5	Yes	No	No	Spam

3 spam emails, 2 not-spam. A new email arrives containing “free” and “invoice” but not “meeting.” Is it spam?

Calculating Prior Probabilities

This is the base rate — how common is each class in your training data?

• P(Spam) = 3/5 = 0.60
• P(Not Spam) = 2/5 = 0.40

Simple counts. No fancy math yet.

Calculating Likelihoods

Now count how often each word appears in each class.

For “free”:

• P(“free” | Spam) = 3/3 = 1.00 (appears in all 3 spam emails)
• P(“free” | Not Spam) = 0/2 = 0.00 (appears in 0 not-spam emails)

For “invoice”:

• P(“invoice” | Spam) = 1/3 = 0.33
• P(“invoice” | Not Spam) = 1/2 = 0.50

For “meeting” (not present in the new email — we use the absence probability):

• P(no “meeting” | Spam) = 3/3 = 1.00
• P(no “meeting” | Not Spam) = 1/2 = 0.50

Making the Prediction

Multiply everything for each class:

P(Spam | new email) ∝ 0.60 × 1.00 × 0.33 × 1.00 = 0.198

P(Not Spam | new email) ∝ 0.40 × 0.00 × 0.50 × 0.50 = 0.000

The model predicts: Spam. And it’s right.

But look at that zero. P(“free” | Not Spam) = 0/2 = 0.00 because “free” never appeared in our not-spam training emails. That zero killed the entire not-spam calculation. Multiplied anything by zero, you get zero. This is a real problem in production.

The Zero Probability Problem — and How Laplace Smoothing Fixes It

This is the problem at a larger scale: you have 10,000 emails in your training dataset. A new email has only one word “cryptocurrency” which has not been seen in your training data. Thus, P(cryptocurrency | Not Spam) = 0. The problem is that your classifier cannot give any prediction for an email with such a word.

The solution is called Laplace smoothing, where 1 is added to all the counts prior to computing the probability. Even if the word is not seen, its count will be 1 and not 0.

The adjusted formula:

P(word | class) = (count of word in class + 1) / (total words in class + vocabulary size)

So P(“free” | Not Spam) becomes: (0 + 1) / (2 + 3) = 0.20 instead of 0.

Not a real probability — but a workable one. The word is still less likely to appear in not-spam than in spam, just not impossibly rare.

In sklearn, this is the alpha parameter. Default is 1.0 (full Laplace smoothing). You can tune it, but starting at 1.0 is almost always fine.

The Three Types of Naive Bayes — Which One Should You Use?

Most tutorials describe all three variants but never tell you how to choose between them. That’s the actually useful part.

Gaussian Naive Bayes

Use when: your features take continuous numeric values – age, height, temperature, test scores, blood pressure.

The assumption: In each class, all the features are normally distributed (bell-curve distribution). The classifier will estimate the mean and standard deviation of all the features of each class based on the training set provided.

Best for: predicting medical diagnoses, sensor classification and other tasks involving features as measurements rather than counts or classes.

When it breaks: Your continuous features are highly skewed or multimodal (bimodal). The assumption of normal distribution is carrying a lot of weight here.

python
from sklearn.naive_bayes import GaussianNB
model = GaussianNB()

Multinomial Naive Bayes

Use when: you need to measure the counts of features – that is, the number of occurrences.

The assumption: Features have a multinomial distribution. For example, the frequency of words in a document – “free” occurred 3 times, “meeting” occurred 0 times, “invoice” occurred 1 time.

Best for: classification of documents, spam detection (when frequency of words is important), classification of news articles, etc. 

When it breaks: when you have negative feature values (Multinomial cannot deal with them) or when features don’t represent counts.

python
from sklearn.naive_bayes import MultinomialNB
model = MultinomialNB(alpha=1.0)  # alpha is Laplace smoothing

Bernoulli Naive Bayes

Use when: your features are binary (each feature either exists [1] or does not exist [0]).

The assumption: features are distributed according to the Bernoulli distribution. How is that different from Multinomial? It takes into account the presence of a word but does not care about its frequency.

Best for: short text categorization and binary features, cases when it doesn’t matter how frequent a word is and only its presence is important (e.g., checking if there is some specific keyword in a text).

When it breaks: when frequency matters. An email can have “free” fourteen times or just once and Bernoulli will treat both emails the same way. It works fine some time and loses important information another.

python
from sklearn.naive_bayes import BernoulliNB
model = BernoulliNB(alpha=1.0)

The Decision Table

Type	Feature Type	Distribution	Best Use Case	When NOT to Use
Gaussian	Continuous numbers	Normal	Medical data, sensor readings	Heavily skewed distributions
Multinomial	Integer counts	Multinomial	Word frequency, TF-IDF	Negative values
Bernoulli	Binary (0/1)	Bernoulli	Word presence/absence	When frequency carries real signal

Complement Naive Bayes — The One People Forget

However, there is a fourth version that most of the guides ignore. The complement naive Bayes (CNB) algorithm is an extension to multinomial which computes probability based on the complement of each class rather than the actual class.

To be more specific, it becomes important when dealing with unbalanced training sets. For example, if 95 percent of the training e-mails are spam while only 5 percent of them are not-spam, the naive Bayes algorithm is going to be biased towards the spam category. CNB algorithm solves this problem as it evaluates “how badly this document fits into the other classes.”

python
from sklearn.naive_bayes import ComplementNB
model = ComplementNB(alpha=1.0)

Use this when your class distribution is uneven. It often outperforms Multinomial on real text classification tasks.

Python Implementation with sklearn (Full Working Code)

Let’s build an actual spam classifier, not just a toy example. This is the part that’s missing from IBM’s article and GeeksforGeeks entirely.

Full Spam Classifier

python
import numpy as np
from sklearn.naive_bayes import MultinomialNB
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.model_selection import train_test_split
from sklearn.metrics import classification_report, confusion_matrix

# Training data -- emails and labels (1 = spam, 0 = not spam)
emails = [
    "free money win prize now",
    "click here free offer limited",
    "win cash prize free entry",
    "meeting at 3pm about project",
    "please review the attached invoice",
    "schedule call tomorrow morning",
    "free free free win money prize",
    "quarterly report attached please review",
    "project update meeting notes",
    "exclusive offer free gift claim now"
]
labels = [1, 1, 1, 0, 0, 0, 1, 0, 0, 1]

# Step 1: Convert text to word count vectors
# CountVectorizer builds the vocabulary and counts word occurrences
vectorizer = CountVectorizer()
X = vectorizer.fit_transform(emails)

# Step 2: Split into train and test sets
X_train, X_test, y_train, y_test = train_test_split(
    X, labels, test_size=0.3, random_state=42
)

# Step 3: Train the classifier
# alpha=1.0 applies Laplace smoothing -- handles unseen words
model = MultinomialNB(alpha=1.0)
model.fit(X_train, y_train)

# Step 4: Predict on test set
y_pred = model.predict(X_test)

# Step 5: Evaluate
print(classification_report(y_test, y_pred,
                            target_names=["Not Spam", "Spam"]))
print("\nConfusion Matrix:")
print(confusion_matrix(y_test, y_pred))

# Step 6: Predict a new email
new_email = ["free prize money claim now"]
new_X = vectorizer.transform(new_email)
prediction = model.predict(new_X)
probability = model.predict_proba(new_X)

print(f"\nPrediction: {'Spam' if prediction[0] == 1 else 'Not Spam'}")
print(f"Confidence: {max(probability[0]):.1%}")

Walk through what this does:

CountVectorizer turns each email into a vector of word counts. The vocabulary is built from training data. “free” gets column 4, “meeting” gets column 9, and so on — every word becomes a number.

MultinomialNB(alpha=1.0) trains the classifier. The alpha=1.0 is Laplace smoothing — it prevents the zero probability problem we saw in the worked example by adding 1 to every word count.

predict_proba() gives you the actual probability scores for each class, not just the winning class. This is useful when you want to set a confidence threshold (“only classify as spam if I’m more than 80% confident”) rather than always picking the top class.

Evaluating Your Model

Accuracy should not be your sole criterion here. In case of spam classification, there are two things which really matter:

Precision — out of all I called spam, what portion of those were really spam? If my precision is low, then this indicates that I am rejecting many legitimate emails.

Recall — out of all the actual spam, what portion was detected by me? If the recall is low, then this indicates that actual spam messages are getting past me. This is also annoying, but in a different way.

F1 score — the harmonic mean of precision and recall. Use this when you care about both errors equally.

Which one is important really depends upon the use case. In case of spam detection, people generally tolerate missing some spam (low recall) better than rejecting legitimate emails (low precision).

The Numerical Underflow Problem — and the Log Fix

Let’s consider an issue that is easy to overlook in small cases but will completely wreck your project in practice.

Multiplying many probabilistic factors in Naive Bayes classifier:

P(spam | email) ∝ P(spam) × P(word₁ | spam) × P(word₂ | spam) × … × P(wordₙ | spam)

Every factor individually is not really big — say, 0.001 or even 0.0003. When multiplied several hundred times, it results in tiny values which floating-point arithmetic interprets as zeros. Python returns 0.0, the comparison doesn’t work, and your classifier fails.

The solution is very simple — take logs on all parts. Since log(a × b) = log(a) + log(b), multiplication changes to addition and there is no more underflow problem.

log P(spam | email) ∝ log P(spam) + Σ log P(wordᵢ | spam)

It’s automatically done by sklearn. Calling the predict() method works in log-probabilities already. You don’t have to do anything yourself. However, it’s good to be aware of that — it will definitely help you when implementing Naive Bayes on your own.

When to Use Naive Bayes — and When Not To

This is the section nobody writes. Let’s fix that.

Use Naive Bayes When…

Your training data is small. Naive Bayes requires estimation of per feature probability distribution, not weight distribution across a million parameters. Naive Bayes learns to a significant extent from a few hundred instances. On the same data set, the neural network will overfit significantly, whereas Naive Bayes does not.

Speed is a hard requirement. Training is all about counting and dividing. Predictions come from multiplying probabilities. In a modern processor, Naive Bayes can classify tens of thousands of documents per second. This has importance for real-time applications, such as fraud detection in payments or moderating live content.

Explainability is required. “The mail was labeled spam due to it containing the following terms: ‘free’ (3× more prevalent in spam than non-spam); ‘prize’ (7× more prevalent in spam than non-spam); and ‘claim’ (5× more prevalent in spam than non-spam).” This is all done by generating an explanation from the probabilities within the model’s tables. Explain the reasoning behind the transformer. 

Feature count is high relative to sample count. Text classification is the quintessential example: 50,000 words in the vocabulary; perhaps 500 examples for training. This is where most machine learning algorithms break down. Naive Bayes works well in such cases since the independence assumption is closer to being true in higher dimensions.

You need a strong, fast baseline. Before you spend weeks tweaking your XGBoost model or finetuning your BERT model, build a Naive Bayes classifier in 10 minutes to see how well it performs. The performance is the benchmark for which all the models must beat. If your complex model cannot beat Naive Bayes significantly, it is a message regarding the problem, not the algorithm.

Don’t Use Naive Bayes When…

Features are strongly correlated. If knowledge of one property gives a good deal of information about another, the assumption of independence has been completely violated – not simply loosely, but such that your probabilities are consistently skewed in an incorrect manner. Data from nearby sensors is one such problem.

Exact probability calibration matters. Naive Bayes classifies classes better than estimating the probabilities of those classes. Predict_proba() output is not a dependable estimate of probabilities. In cases where you want to feed the output into a decision-making process, use logistic regression or calibrate Naive Bayes’ output using some other technique.

You have large, rich labeled datasets and accuracy is the only goal. With enough data, gradient boosting (XGBoost, LightGBM) and trained language models will beat Naive Bayes in accuracy. Efficiency and ease of use don’t matter as much when your goal is to do an offline batch process of millions of examples.

Naive Bayes vs. Logistic Regression — The Practical Comparison

These two come up together constantly. Here’s when each wins:

Criterion	Naive Bayes	Logistic Regression
Training speed	Much faster	Slower (iterative optimization)
Small datasets	Better	Can overfit
Correlated features	Hurts badly	Handles them
Probability calibration	Poor	Much better
Interpretability	High	Moderate
Online learning	Easy to update	Possible but messier

The short answer: use Naive Bayes first. Switch to logistic regression if you need better probability estimates, have correlated features, or have enough data to justify the extra training time.

Is Naive Bayes Still Relevant in 2026?

Yes. Specifically for these scenarios:

Edge devices and embedded systems. In cases when it just won’t fit in memory or when there are latency issues. A spam detector implemented in a router, an intent classifier in a smartwatch — Naive Bayes occupies a few kilobytes in memory and takes microseconds to react.

Streaming real-time classification at high throughput. When you’re classifying 100,000 transactions per second, the model needs to be fast enough to keep up. Naive Bayes is.

Extremely limited labeled data. The performance gap between Naive Bayes and a highly optimized language model rapidly shrinks with small amounts of labeled data. With just 200 training examples, they tend to perform similarly.

Explainability-first requirements. Explainability of a model’s decision is becoming an explicit requirement in financial services, healthcare, and legal technologies. Naive Bayes does this inherently, while transformers do not.

Strangely, the advent of LLMs has actually increased relevance of Naive Bayes in some ways since it became clear which problems LLMs can’t solve (explainability, low latency, small dataset).

Real-World Applications — Beyond the Textbook

Most articles list applications in one sentence each. Let’s actually look at how this works in practice.

Spam filtering. The popular open-source spam detection software SpamAssassin, which is still running on many mail servers today, is based on probability estimates used in Naive Bayes. The insight the authors of SpamAssassin had: you do not need a highly precise classifier, you need a fast enough and reasonably accurate enough classifier for filtering emails. This way, Naive Bayes worked in the 1990s and continued working until recently.

Sentiment analysis at scale. Multinomial Naive Bayes classifier of sentiments on product reviews into positive, negative, and neutral categories processes tens of thousands of reviews per minute on CPU alone. In some use-cases of e-commerce, it’s better than transformer-based sentiment classification due to the latency and the slight difference in precision (2-3 percent).

Medical diagnosis support. Gaussian Naive Bayes estimates the probability of diseases based on symptoms using small-size clinical databases where you may have 300 labeled patients, not 300,000. The Naive Bayes algorithm does not require huge databases to learn meaningful information about medical features such as age and blood pressure readings.

Historical NLP: The Federalist Papers. In 1963, Mosteller and Wallace applied the Bayesian approach for distinguishing the authorship of the controversial Federalist Papers between Alexander Hamilton and James Madison using word frequency analysis. This is one of the early examples of applying the probabilistic classification method to text. The assumption of independence made by the researchers, needless to say, was wrong. However, the conclusions reached were verified decades later by another technique and proved to be accurate.

Cybersecurity / network intrusion detection. Naive Bayes classifies network packets in the context of a streaming environment where the cost of latency of more complex models would be unacceptable. Here features of each packet (protocol, port, size of the packet, flags in the packet’s header) are considered independent variables.

Advantages and Disadvantages

Advantages

Fast to train. No iterative optimization, no gradient descent. Training is one pass through the data to count occurrences and estimate probabilities.

Works well with small datasets. The assumption of independence is actually an advantage since we have less parameters to estimate and thus less data to estimate them effectively.

Handles high-dimensional data. Not a problem to work with 100,000-word vocabulary since the probability of each word is calculated independently.

Easy to update. New training data? Add the counts. You don’t have to retrain from scratch. This makes Naive Bayes one of the few classifiers that supports genuine online learning with minimal overhead.

Naturally multi-class. The algorithm works with any number of classes without modification because you just calculate the posterior of each class and choose the largest one.

Disadvantages

The independence assumption. This assumption is violated in problematic ways when features are indeed correlated. The model will still make its predictions, and it will just do so in an off-by-specific-margins way.

Poor probability calibration. Naive Bayes works well for ranking, but does not estimate probabilities well at all. Values returned by predict_proba() tend to be close to zero and one, much higher than they should be. To get calibrated probabilities, use Platt scaling or isotonic regression.

Zero-frequency problem. Solved with Laplace smoothing in practice, but useful to know if you’re debugging.

Assumes equal importance of features. All features have the same weight in the computations. In real life, not all features carry equally much information. Logistic regression models and similar techniques learn feature importances directly, while Naive Bayes doesn’t.

FAQs

Where This Leaves You

Naive Bayes is a 260-year-old algorithm powering the classification engine of your email client right now. This isn’t an appeal to nostalgia; it’s an indicator of the algorithm’s continued usefulness because of its ability to solve very specific problems that no other algorithm solves as well: small data, fast execution, interpretability, and high-dimensional text.

While this algorithm will not outperform a carefully tuned language model in terms of accuracy if there are millions of labeled instances and time to train, it wasn’t meant to solve this problem.

The single most common error I’ve seen made by those building their first machine learning systems is going to sophisticated algorithms without an appreciation of what simple algorithms can do. First, try running Naive Bayes on your problem. See how accurate it is. Then figure out if it makes sense to pay the price in training time, inference latency, memory usage, and interpretability for a more sophisticated algorithm.

Sometimes, it does make sense. Sometimes, however, the 18th-century clergyman’s formula is all you need, and you’ll get your algorithm into production before the GPU cluster finishes booting up.