Type I and Type II Errors

Just because you’ve gotten a significant finding does not mean you have discovered something.

That’s what they don’t tell you straight out. The trouble is, you could end up with a false positive or a false negative either way, depending on whether you want to find an effect that doesn’t exist or mask an effect that does. While knowing the terms is important, knowing which one you need to guard against is where the knowledge comes in handy.

The following will cover both types of errors, the statistical relationship between them, examples from clinical science, A/B testing, and machine learning, and how you can choose which error is more important before you even do anything.

What Are Type I and Type II Errors?

Actually, these concepts have been simplified more by most authors than made complicated.

A Type I error refers to a false positive, whereby there was a rejection of a hypothesis that would have proven to be true. It can be stated simply that a non-existent effect was observed.

A Type II error is a false negative whereby one fails to reject a hypothesis that would have proven to be true in reality. It means that a significant effect was not detected.

It should be noted that this happens because every statistical test involves taking samples from the whole population, which is always subject to variations. These variations may sometimes mimic actual effects when they are nothing but noise, while at other times they may mask actual effects.

The Simple Way to Remember Which Is Which

Consider, for example, the operation of a smoke detector.

In the case of a Type I error, the detector will activate when there isn’t a fire – disruptive, wasteful, erodes faith in the system.

In the case of a Type II error, the detector will not alert when there really is a fire – missed the event.

It depends entirely upon what you’re testing, which is the whole idea behind the decision framework here.

Type I Error — Definition, Alpha, and the Peeking Problem

The key point here is that committing a Type I error is not a matter of pure chance or being extremely unlucky. This probability is something you decide before performing a statistical test.

The probability of making a Type I error is also called the significance level and denoted by α. Thus, setting α = 0.05 means you accept the 5% chance of committing a type I error. Again, note that the alpha does not mean you have a 5% chance of getting a false result overall, but rather a 5% chance of getting a false positive result if the null hypothesis is indeed true.

Note that 0.05 is the standard value chosen due to convenience, but depending on the context, other thresholds can be selected. For instance, particle physicists need five sigma threshold (roughly 0.0000003) to call something a discovery. Similarly, FDA sets the bar even higher for clinical trials with the total alpha equal to 0.05 per side.

There are many other ways one could commit a Type I error, one of them being peeking during the study itself. Say you perform an A/B testing and check the results daily until you achieve the required level of significance.The p-value shifts as data accumulates; stopping early locks in a snapshot that may not hold.

Type I Error Examples Across Three Domains

Clinical research: Early-phase studies suggested that a particular drug lowered blood pressure, with p-value = 0.04. The researchers published their findings. In a later study with a larger number of participants, the drug was found to produce no effect – the initial finding was just noise. The patients were prescribed a drug that had no therapeutic effect; meanwhile, funds that would have been able to find more promising treatment approaches were squandered.

A/B testing: A growth team ran an experiment with a new checkout button color for five days and noticed a 14% increase in conversion rate. Two weeks after launch, the conversion rate dropped back to baseline levels. The test ended before meeting the necessary sample size criteria; what was observed was just a random blip in data. Here we see the peeking issue being put into practice.

Machine learning: A spam filter misclassifies emails from the user’s work domain as spam. Any such error represents a Type I mistake. For most people, a Type I error is far more undesirable in this case than a Type II error – ignoring spam is fine, but missing an important business email isn’t.

Type II Error — Definition, Beta, and the Underpowering Problem

Type II errors occur where there is actually something but the test does not find it. So the experiment is conducted, there is no significant result, and the conclusion drawn that it did not work, despite being incorrect.

The probability of making a Type II error is denoted by beta (β). Power in statistics is defined as 1 – β, which means the chance that a test will identify an effect when it exists. The work done by Jacob Cohen in 1988 on statistical power set the convention of 80% being adequate for practical research in behavior without being mathematically proven but recognizing the meaningfulness of the 20% Type II error rate.

However, in practice, many studies are well below this standard. According to a meta-analysis published in Nature Reviews Neuroscience by Button et al. in 2013, the median power of studies in neuroscience research was approximately 21%. Thus, the results of these studies failed to identify effects in about 67% of all cases. Such underpowering of studies lies at the heart of the replication crisis, where studies that could not be replicated were not faked – they were simply underpowered.

Low power almost always comes from one place: sample sizes chosen by budget or convenience, not by calculation.

Type II Error Examples Across Three Domains

Clinical research: The clinical trial in oncology was performed using only a few dozen patients, while needing 250 patients to detect the effect with 80% statistical power. As a result, the study reported a lack of any statistically significant survival benefits from the treatment and stopped enrolling additional patients. In an additional well-powered study, the effect was detected, but it was already too late for most of the patients who were denied access.

A/B testing: The product managers were performing a test for three days with 600 visitors per variation in order to determine whether the new checkout design improved customer conversions. At the true uplift level of 7%, the power of 80% would be achieved when there are about 3,800 visits per variant. No statistically significant uplift was seen and the redesign was abandoned.

Machine learning: The fraud detection algorithm fails to detect fraudulent transactions with 28% probability. This is a Type II error. It can usually happen more often than the false positive (Type I error) due to the high costs of making a wrong decision in this context.

The Trade-Off — Why Reducing One Error Always Increases the Other

Surprisingly, both mistakes tend to be treated by individuals separately, as two different issues that can be solved independently. Not so.

Reducing alpha level, for instance, from 0.05 to 0.01, decreases the probability of getting a false-positive result. However, at the same time, it lowers the statistical power and increases beta-level – meaning that your tests become less sensitive and, at the same time, more conservative.

It works in the opposite way as well: increasing the statistical power through enlarging the sample size lowers the number of Type II errors. On the other hand, lowering the significance threshold in order to reduce costs will lead to an increase in Type I errors.

The point that remains hidden by many guides is the existence of the shared area between the distributions of null and alternative hypotheses in which both types of errors coexist, making any change in the decision criterion affect the balance between them.

That said, sample size remains the only variable that allows reducing both errors simultaneously; any changes in other variables create a direct trade-off: increasing alpha to 0.10 will make tests more sensitive yet raise the number of Type I errors.

So before choosing a threshold: what does each type of wrong actually cost you?

Which Error Is Worse? A Domain-by-Domain Answer

The truth is that all articles try to solve this problem in some way. However, since the answer is given on a generic level, the problem still exists, and we can only scratch the surface of its meaning.

For research in medicine and clinical studies: Type I error is a more critical problem in most cases. Approving an ineffective or dangerous medicine can lead to undesirable consequences. That is why, in clinical trials, it is crucial not to miss something; the risk should not be more than 0.025 per arm in two-sided confirmation trials. It changes in rare diseases because missing treatable diseases will mean a life lost forever whereas giving a false positive means further testing.

A/B testing and conversion optimization: Type II is an error that costs money. All companies are obsessed with false positives while working with low statistical power in their tests. It is absolutely wrong and can cost quite a lot. When testing 50 hypotheses annually and having 40% power, one risks losing 30 of them if 80% of power was used. And it adds up every cycle of testing; it is not an imaginary value but real money lost on a missed improvement of 8% conversion rates.

Legally speaking: Criminal proceedings actually put emphasis on trying to avoid committing Type I error, that is sentencing someone who is innocent. The standard for proving guilt “beyond reasonable doubt” is quite high, and rejecting the null hypothesis of innocence thus becomes quite hard. Guilty individuals getting acquitted, a Type II error, is considered a lesser consequence. This is an intentional choice made by society, and reflected in the threshold.

The logic: Before choosing your alpha and running a power analysis, determine what will happen when you commit an error either way. Try quantifying those costs, be it financial, patients’ welfare, or your team’s reputation. Then, whichever number you end up with will tell you which type of error to guard against first.

Type I and Type II Errors in Machine Learning

In my view, the way machine learning treats those mistakes will make them much easier to spot than the approach taken by classical statistics – simply due to their immediacy and the existing terminology.

Each classification algorithm can give rise to four events, arranged in the confusion matrix format: true positives, true negatives, false positives (or Type I error), and false negatives (or Type II error). There are two scores that lie on top of these categories.

Precision is a ratio of correct positive classifications. When precision is high, there will be less Type I error, which means that if the prediction was positive, it probably did not happen by accident. For example, the system which is 99% precise will very seldom mistake a non-spam email for spam.

Recall is the proportion of actual positive classifications caught. Low recall means a high number of Type II errors, or missed events. The model with higher recall will detect more cancer cases but will require more follow-up procedures for healthy people as well.

It’s not possible to optimize both, and that’s an inherent constraint, not just for one algorithm but due to math itself. If you’re optimizing for maximum recall, you’ll get false positives for legitimate emails. If you’re optimizing for maximum precision, you’ll miss things in the screening for cancer. The F1 score provides a compromise between the two, but ultimately, you must decide which one you’re going to optimize for because it’s not a math question; it’s a domain question.

How to Minimize Type I and Type II Errors

Choose alpha prior to looking at any data. Once you have the outcome, your judgment is now biased toward your data. Specifying your significance level in advance, before seeing any data, eliminates the temptation of moving your target post hoc. It is required for clinical trials; few A/B testing teams do so.

Do a power analysis prior to starting the test. Choose your expected effect size, desired power (at least 80%), and alpha level, and calculate your needed sample size. G*Power is a free, open-source software that can perform power analysis for most common tests. You can perform it in Python using the statsmodels.stats.power package. Perform a power analysis prior to running, not afterwards, when the result doesn’t look good.

Go with the sample size that came out of the power analysis. Exactly. This is your only way to simultaneously decrease both errors. If you choose your sample size based on a cost limitation rather than the required power, your analysis is no longer trustworthy no matter what side the coin lands on.

Don’t stop A/B tests early. Set a fixed runtime before launch; stick to it. If your team legitimately needs to check interim results, use sequential testing methods that adjust the significance threshold to account for multiple looks — these are purpose-built for that problem and keep your actual false positive rate honest.

Industry-Specific Alpha Thresholds

Different fields set different bars for what counts as sufficient evidence — and the variation is enormous.

Field	Standard Alpha	Why
Social science / behavioral research	0.05	Cohen’s convention; widely adopted since the 1960s
FDA drug approval — confirmatory trials	0.025 per arm	Two-sided testing requirement; patient safety standard
Particle physics	~0.0000003 (5-sigma)	Discoveries require near-certainty before publication
Web experimentation / CRO	0.05, often sequential	High test volume; sequential methods preserve error control across many looks
Genomics	~5 × 10⁻⁸	Bonferroni-adjusted for millions of simultaneous genetic markers tested

Frequently Asked Questions