P-value

Many systematic reviews show that p-values have been consistently calculated and interpreted incorrectly — even by those who are using them in published studies. 

That’s our baseline. Not to make anyone look bad, but just to let you know that there’s a problem with some of the conventional wisdom out there.

Typical courses cover what a p-value is. But few cover what it isn’t, which is crucial information. Because it’s mistakes about this that get published and not replicated.

What Is a P-Value?

The p-value is the probability of obtaining at least as extreme results as the ones you obtained, given that the null hypothesis is true.

And all of this comes down to that one sentence. The question is not whether your hypothesis is right; it is, if everything you think is going on actually isn’t, how likely would it be for random chance alone to generate numbers like these?

This system was invented by Ronald Fisher in the early 1920s to serve as a simple criterion for experiments in agriculture. The 0.05 figure was just a convenient convention, not an absolute rule.

The Plain-English Version

Suppose you flip a coin ten times and receive nine heads. You suspect foul play. The p-value asks: If the coin is an entirely fair one, what is the probability of receiving nine or more heads out of those ten flips?

Around 1.1%.

Therefore, either something strange occurred using a fair coin, or you do not have a fair coin. The p-value does not reveal whether that is the case.

What P-Value Actually Tells You — and What It Doesn’t

Here’s the kicker — this is where almost every single textbook glosses over the point.

A significance level of 0.03 doesn’t tell us there is a 97% chance that our hypothesis is right. It certainly doesn’t indicate that our null hypothesis is probably wrong. Neither does it say anything about the probability of repeating this experiment with similar results.

This significance level means one thing only: if our null hypothesis were true, we should expect to get these results 3% of the time.

The American Statistical Association went as far as publishing an official paper on this issue in 2016, stating, in no uncertain terms, that ‘p-values are commonly misused and misinterpreted.’ This statement came out after several months of discussion among a panel of twenty experts. But the confusion continues.

P-value IS	P-value IS NOT
Probability of your data, given H₀ is true	Probability that your hypothesis is true
A measure of how surprising your results are	Proof that an effect exists
A reason to investigate further	A verdict

The Misconception That Breaks Research

Whenever the finding p < 0.05 is interpreted as “the theory is most likely correct,” critical questioning ceases. Researchers go on to publish. Others attempt replication. The effect evaporates.

It’s not evidence. Just an indication.

Such an occurrence is among the identified contributors to the replication crisis — the process whereby we find out that many of the results produced by research into psychology, medicine, and nutrition science are unreplicable. It’s not the only contributor to this problem. But it is a major one.

How to Calculate a P-Value (Step by Step)

Every statistical test follows the same five stages, regardless of the specific method.

Step 1 — State your null hypothesis (H₀) Define the baseline assumption: nothing is happening. Example — “There is no difference in exam scores between students who studied with music versus without.”

Step 2 — Define your alternative hypothesis (H₁) What you are actually testing. “Students who studied without music score higher.” This step also determines whether you need a one-tailed or two-tailed test.

Step 3 — Select your test and calculate the test statistic The test statistic measures how far your data sits from what H₀ predicts. Different tests produce different statistics — a t-value, a z-score, an F-ratio, a chi-square value.

Step 4 — Convert the test statistic to a p-value In practice, software handles this step. In Python:

python

from scipy import stats

t_stat, p_value = stats.ttest_ind(group_a, group_b)

print(p_value)

Step 5 — Compare p to your significance threshold (α) If p ≤ α, reject H₀. If p > α, you don’t reject it. The default α = 0.05 is a convention, not a universal rule; in clinical trials, 0.01 or tighter is standard.

Which Statistical Test Do You Need?

Your situation	Use this test
One group mean vs. a known value	One-sample t-test
Two independent group means	Two-sample t-test
Same subjects measured twice	Paired t-test
Large sample, known population variance	Z-test
Association between two categorical variables	Chi-square test
Comparing three or more group means	F-test (ANOVA)
Linear relationship between two continuous variables	Pearson correlation

One-Tailed vs. Two-Tailed P-Value

A two-tailed test asks whether a difference exists in either direction. A one-tailed test asks whether the difference goes in a specific direction you predicted before collecting data.

One-tailed p-value = two-tailed p-value ÷ 2 — but only when the effect runs in the predicted direction. If it goes the opposite way, the one-tailed p-value becomes 1 minus that result. Large, as it should be. You got a surprising outcome in the wrong direction; the p-value reflects that accurately.

How to Interpret Your P-Value

P-value	What it signals
< 0.001	Very strong evidence against H₀
0.001 to 0.01	Strong evidence against H₀
0.01 to 0.05	Moderate evidence against H₀
0.05 to 0.10	Weak, borderline — requires judgment
> 0.10	Insufficient evidence to reject H₀

Honestly, the 0.05 threshold deserves more scrutiny than it usually gets. Fisher reportedly chose it because it corresponded roughly to two standard deviations from the mean under a normal distribution — a round, convenient number rather than a derived truth. High-stakes fields like pharmaceutical trials or aviation safety typically use 0.01 or stricter.

What to Do When p > 0.05

But failing to reject H₀ is not the same as confirming it.

A p-value of 0.11 doesn’t mean nothing is happening. It means your data isn’t unusual enough — given your current sample — to rule out chance. The effect might be real; your study might simply lack the statistical power to detect it.

When p > 0.05, the right next steps are:

• Check whether your sample was large enough to detect the effect size you expected
• Calculate statistical power — if it falls below 0.80, your test was likely underpowered before you even started
• Examine the effect size estimate alongside the p-value, not as an afterthought
• Consider replication with a larger dataset before drawing any conclusion about H₀

P-Value vs Effect Size — Why Significance Doesn’t Mean Importance

Most articles on this topic get this exactly wrong: they treat a statistically significant p-value as the goal. It isn’t.

I’ve seen analysis reports — and published papers — where the authors reported p < 0.001 and concluded the intervention “works,” with no mention of how large the effect actually was. Consider a study of 80,000 participants that finds a wellness app improves average sleep duration by 4 minutes per night, p < 0.0001. Statistically significant. Practically useless. Nobody’s life changes because of 4 extra minutes.

Effect size answers the question the p-value ignores: how big is the difference?

Cohen’s d is the standard measure when comparing two group means. A d of 0.2 is considered small, 0.5 medium, 0.8 large — though Cohen himself called these rough guidelines, not universal rules. The APA Publication Manual — current edition — recommends reporting effect size alongside any inferential statistic. Many peer-reviewed journals now require it as a condition of publication.

Factors That Affect Your P-Value

Four variables move your p-value — independently of whether the underlying effect is real.

Sample size. Larger samples produce smaller p-values, all else equal. Add enough participants and almost any trivial difference becomes statistically detectable. This is not a flaw in the math; it’s a reason to always pair p-values with effect sizes.

Effect size. A larger real-world difference produces a smaller p-value. Which sounds obvious until it isn’t — because researchers sometimes increase sample size specifically to force a small effect past the 0.05 line.

Data variability. High variance makes signals harder to separate from noise; noisier measurements push p-values higher, even when the underlying effect is genuine.

Violated test assumptions. Every statistical test assumes something about your data — normality, independence, equal variances across groups. Break those assumptions and the p-value becomes unreliable regardless of its size.

How to Report a P-Value Correctly

Students get this wrong constantly, and so do some reviewers.

The APA format rules:

• Lowercase, italicized p
• No leading zero: write p = .047, not p = 0.047
• For very small values: p < .001, not p = .000 or p = 0.00001
• Report exact values whenever possible rather than “significant” or “n.s.”

Correct: “The intervention improved recall performance, t(58) = 3.41, p = .001, d = 0.74.”

Incorrect: “Results were significant (p<0.05).”

The second version tells a reader almost nothing about the actual evidence. It’s the reporting equivalent of a restaurant critic writing “the food was good” — no dish named, no standard applied, nothing a reader can actually use.

Common Mistakes When Using P-Values

Treating p < 0.05 as proof. It’s a probabilistic signal. A different sample from the same population might produce p = 0.08. The threshold is a convention; treat it as one.

P-hacking. Run 20 independent tests at α = 0.05 and you expect one false positive by chance alone — that’s just the math. Researchers who test repeatedly until significance appears are exploiting this, deliberately or not. The Bonferroni correction fixes it: divide your threshold by the number of comparisons. Testing 10 hypotheses simultaneously means your adjusted threshold is 0.005, not 0.05.

Ignoring effect size. Statistical significance and practical importance are separate questions. A significant p-value for a negligible effect is a finding that barely matters outside the paper it’s published in.

Reading p > 0.05 as proof H₀ is true. It isn’t. Absence of evidence is not evidence of absence… and that principle applies more precisely to p-values than almost anywhere else it gets quoted.

Frequently Asked Questions

Closing

The p-value was designed to flag when data is surprising enough to take seriously. It was not designed to be a pass/fail test for truth.

Pair it with effect size. Check your statistical power before you run the analysis, not after. Report the exact value in the correct format. And when the result is borderline, sit with that uncertainty rather than rounding up to a conclusion.

The p-value does not tell you whether you’re right. It tells you whether your data is worth talking about. That’s a narrower claim than most people want from it — but it’s an honest one.