T-Test Explained

Most people learn the t-test formula. Almost nobody learns when the result actually means something. That gap (between running a test and trusting the answer) is what this article closes.

What Is a T-Test? (The One-Sentence Version)

The t-test determines whether the difference in mean value between two groups is significant or insignificant. The test is done using a statistical measure called the t-value.

This is defined by the following formula:

t-value = mean difference / standard error

Signal-to-noise ratio. The mean difference between groups represents the signal; i.e., what you have found. Standard error represents the noise – the variance that occurs within a given sample irrespective of there being any significant difference within the population. Therefore, a high t-value indicates a clear signal. A low t-value suggests that the signal cannot be distinguished from the noise.

Gosset developed this statistic while working as a mathematician for the Guinness Brewery in Dublin in 1908. His employers only allowed him to publish results on the condition that he did not use any data collected by the firm or put forward his true name, and thus he adopted the pseudonym “Student.”

A statistical method like the t-test ensures you know the difference between an effective drug and one that simply happened to perform well in your sample of 40 patients.

The 3 Types of T-Test — and When Each One Applies

Test Type	Use When	Example Scenario
One-Sample	Comparing your sample mean against a known reference value	Does this production batch hit the claimed 500mg dosage?
Independent Samples	Comparing two separate, unrelated groups	Do engineers and marketers earn different average salaries?
Paired Samples	Comparing the same group measured twice	Did blood pressure drop after starting the medication?

One-Sample T-Test

You have one sample. One reference figure. You are wondering whether your sample deviates significantly from the given reference.

The claim made by the dietitian is that his protein supplement is exactly 25 grams per serving. In a lab experiment, you test 20 servings and find an average of 22.3 grams. Was there really a deficit? Or does the difference arise from measurement error? This is where the one-sample t-test finds its use.

Formula:

t = (mean of sample – reference figure) / (SD / √n)

In essence, each of the variables in the formula means something. The numerator represents the deviation of your sample values from the expected value. The denominator normalizes the deviation based on the sampling variability. Larger sample sizes lead to smaller standard errors.

Independent Samples T-Test (Two-Sample T-Test)

Two samples. Two different people in each sample. No relation between them.

This is where everyone makes the same crucial error – making an assumption that both samples have the same variance, which is not always the case. In this situation, you use the Welch t-test rather than the Student’s test that requires almost equal variances in both samples.

And this is where no one tells you what modern statisticians use anyway – they all prefer Welch’s test for any situation because of how well it performs, no matter the variances being compared. Python’s scipy.stats.ttest_ind() runs Welch’s automatically with equal_var=False. R’s t.test() defaults to it.

Paired Samples T-Test

Same group. Two tests. One difference between them.

The trick is realizing that you aren’t comparing two different groups of scores. You are trying to find the difference within each pair. Ten people provide their weights prior to beginning the diet, and then post-diet. Each of these participants’ weights is subtracted from one another, then you run the average of their differences against zero.

This particular method is statistically stronger than the independent t-test where applicable because testing the same individual twice means the variance is removed from the equation. Two people who might start at vastly different weights might both respond similarly to the same stimuli.

Quick Decision Guide — Which T-Test Should You Use?

Answer these three questions in sequence; stop once you get an answer.

1. Are you comparing your sample mean against a fixed known number? Yes. One-sample t-test.
2. Are you measuring the same individuals twice? Yes. Paired samples t-test.
3. Do your two independent groups have similar variance? Yes. Independent samples t-test (Student’s). No. Welch’s t-test.

T-Test Assumptions — And How to Actually Check Them

Every statistics guide lists these assumptions. Almost none explain what to actually do with them.

Normality

A normal distribution is required for your data from the population. For a large sample size (n > 30), this assumption doesn’t matter much due to the application of the Central Limit Theorem, which makes the distribution normal irrespective of the initial one. However, on small samples, this becomes critical.

How to check it (Shapiro-Wilk test):

from scipy import stats

stat, p = stats.shapiro(your_data)
print(f"Shapiro-Wilk p-value: {p:.4f}")

# P < 0.05 indicates the rejection of normality. Choose a non-parametric option accordingly.

In case of rejection of normality on a small sample, move away from using t-test to its non-parametric counterpart.

Homogeneity of Variance

For independent samples tests: the spread of values in both groups should be broadly similar.

How to check it (Levene’s test):

stat, p = stats.levene(group1, group2)
print(f"Levene's test p-value: {p:.4f}")

# p < 0.05 means unequal variances. Use Welch’s t-test

Or use Welch’s by default and bypass this check entirely. That’s the more practical path. 

Independence and Scale

Each observation should come from a different individual (in the case of independent samples). The measuring variable is expected to be quantitative – weight, time, score, or blood pressure. It cannot be categorical in nature. It cannot be binary either.

How to Calculate the T-Value

The formula changes by test type. The logic doesn’t: signal over noise.

One-Sample:

t = (sample_mean – reference) / (std_dev / sqrt(n))

Paired Samples:

t = mean_of_differences / (std_of_differences / sqrt(n))

Independent Samples (Welch’s):

t = (mean1 – mean2) / sqrt((var1/n1) + (var2/n2))

As the amount of variation increases in the groups or the number of cases decreases within them, the denominator gets larger. With an increase in noise, there will be less t-value, making it difficult to distinguish the actual difference. It is for this reason that high-variation small-studies fail to capture differences that exist.

How to Run a T-Test in Python

from scipy import stats
import numpy as np

# ONE-SAMPLE T-TEST
# Does the protein powder hit its 25g-per-scoop claim?
measurements = [22.1, 23.4, 21.8, 24.0, 22.7, 23.1, 21.5, 22.9, 23.6, 22.3]
reference    = 25.0

t_stat, p_val = stats.ttest_1samp(measurements, reference)
print(f"One-Sample  | t = {t_stat:.3f}, p = {p_val:.4f}")


# INDEPENDENT SAMPLES T-TEST (Welch's)
# Do two teaching methods produce different exam scores?
method_a = [78, 82, 85, 79, 88, 84, 90, 76, 83, 87]
method_b = [91, 85, 93, 89, 87, 94, 90, 88, 92, 86]

t_stat, p_val = stats.ttest_ind(method_a, method_b, equal_var=False)  # Welch's
print(f"Independent | t = {t_stat:.3f}, p = {p_val:.4f}")


# PAIRED SAMPLES T-TEST
# Did blood pressure drop after 8 weeks of exercise?
before = [145, 138, 152, 140, 148, 135, 155, 142, 149, 137]
after  = [132, 128, 141, 130, 135, 125, 143, 131, 138, 126]

t_stat, p_val = stats.ttest_rel(before, after)
print(f"Paired      | t = {t_stat:.3f}, p = {p_val:.4f}")


# CHECK NORMALITY FIRST
stat, p = stats.shapiro(method_a)
print(f"Shapiro-Wilk (Method A): p = {p:.4f}")

# CHECK VARIANCE EQUALITY
stat, p = stats.levene(method_a, method_b)
print(f"Levene's Test: p = {p:.4f}")

Notice the equal_var=False argument in ttest_ind(). That single flag is what switches it to Welch’s. Leave it at default True and you’re running Student’s version, often without knowing it. If your groups have meaningfully different variances, Student’s version will give you inflated Type I error rates; that’s a problem worth accounting for.

How to Interpret Your T-Test Results

What the T-Value Tells You

T-value in itself does not mark an end to your analysis. The t-value is just a step to finding out the p-value. Higher t-values indicate that there is a lot of difference between your samples compared to the noise present. What ‘a lot’ means is dependent on your sample size and degrees of freedom.

What the P-Value Actually Means (And What It Doesn’t)

In fact, that is where most explanations about t-tests fall flat.

The p-value doesn’t mean the probability of your result being real. What it really means is: if the null hypothesis was true (meaning there is indeed no difference in the population), how likely would an observed value be at least that extreme due to random variation alone?

p = 0.03 means: under the assumption that there is no difference, this observation happens purely due to random sampling just 3% of the time – below the traditional 0.05 threshold, which means you reject the null hypothesis.

What p < 0.05 doesn’t mean:

• There’s a 95% probability the difference is real
• The difference is big enough or impactful
• It holds true for the next experiment too

A formal statement regarding the improper use of p-values was made by the American Statistical Association in 2016, followed in 2019 by an editorial published in the journal The American Statistician, encouraging scientists to give up on using the p-value of 0.05 as the point of decision. In one word, the basic idea was that statistically significant findings should be used merely as a basis for further interpretation.

Statistical Significance vs Practical Significance: Effect Size

Almost all articles addressing this subject make this mistake. Significant p values mean that the outcome was significant. Not really.

With 10,000 people in your sample, a p < 0.001 could be calculated from a difference of 0.2 out of 100. Statistically highly significant. Insignificant from the point of practical significance. A large sample size allows you to test for highly insignificant effects because statistically these are real effects but too small to do anything about.

That’s why Cohen’s d should be used. The standardized effect size for t-test outcomes.

Cohen’s d = (mean1 – mean2) / pooled standard deviation

Benchmarks from Jacob Cohen’s framework: 

• d = 0.2 is a small effect
• d = 0.5 is a medium effect
• d = 0.8 is a large effect

def cohens_d(group1, group2):

    n1, n2     = len(group1), len(group2)

    var1, var2 = np.var(group1, ddof=1), np.var(group2, ddof=1)

    pooled_std = np.sqrt(((n1 – 1) * var1 + (n2 – 1) * var2) / (n1 + n2 – 2))

    return (np.mean(group1) – np.mean(group2)) / pooled_std

d = cohens_d(method_a, method_b)

print(f”Cohen’s d = {d:.3f}”)

Run the t-test. Then calculate Cohen’s d. Both numbers together tell you whether the result is real and whether it matters. One without the other is half an answer.

How to Report T-Test Results (APA Format)

No competitor article covers this. But it’s exactly what researchers, students, and practitioners need the moment they have output in hand.

Standard APA format:

t(degrees of freedom) = t-value, p = p-value, d = Cohen’s d

Worked example (Independent samples):

“Participants using Method B scored significantly higher than those using Method A, t(18) = -3.42, p = .003, d = 0.89.”

Worked example (Paired samples):

“Systolic blood pressure decreased significantly after the 8-week program, t(9) = 8.74, p < .001, d = 2.76.”

Three guidelines for reporting results: report the exact p-value instead of “p < .05”; place the degrees of freedom in parentheses immediately after the t; and finally, report Cohen’s d. All this information must be present in one sentence for a reader to discern its significance and importance.

One-Tailed vs Two-Tailed Tests — The Practical Rule

A two-tailed test asks, “Is there a difference, either way?” while a one-tailed test asks, “Is there a difference in precisely this direction?”

Default to two-tailed.

One-tailed testing is valid only if you made a very strong directional prediction beforehand and have absolutely no realistic scenario under which the result would go the other way. It becomes p-hacking if you decide whether to use one-tailed or two-tailed based on whether the data went the way you wanted. In that case, you’re applying an inappropriate statistical threshold.

scipy computes two-tailed p-values by default. Divide by 2 if you need a one-tailed p-value. But be sure it’s really appropriate.

When NOT to Use a T-Test

• More than two groups: ANOVA is needed. When comparing three sets of results using three t-tests, there is an increase in the Type I error rate.
• Ordinal or categorical data: The t-test is used to compare means. For this reason, a continuous distribution is required. Likert Scale or Yes/No data should use the Chi-Square or Mann-Whitney test.
• Severely non-normal data with small sample size: The Central Limit Theorem won’t help you when your sample size is less than 15. Try using a non-parametric test.
• Multiple dependent variables: MANOVA should be used rather than multiple t-tests.

Non-Parametric Alternatives

However, when normality cannot be assumed and when your sample size is small, there are two tests that have the same function as the t-test without needing a normal distribution:

The Mann-Whitney U test takes over from the independent samples t-test and compares the distributions of the two groups using ranks rather than comparing means.

The Wilcoxon signed-rank test takes over from the dependent samples t-test.

# Non-parametric replacements

stats.mannwhitneyu(group1, group2, alternative=’two-sided’)

stats.wilcoxon(before, after)

Common T-Test Mistakes

1. Using independent when the data is actually paired. When the same participants appear in both conditions, using independent analysis will lead to an incorrect calculation of the standard error and increase your likelihood of missing an effect. This is not an insignificant issue; it will influence your conclusion.
2. Treating p = 0.049 as success and p = 0.051 as a failure. In reality, there is only a minuscule difference between these numbers – the boundary line is an arbitrary construct. A significant effect that gives p < 0.051 and Cohen’s d = 0.8 carries a lot more meaning than one where p < 0.003 and d = 0.1. Report your exact statistics and effect size.
3. Doing multiple t-tests instead of ANOVA. By convention, each of these tests has a 5% probability of producing a false positive. Run 20 t-tests on the same sample, and you will receive about one false positive simply because of random variation. Do an ANOVA and post hoc tests.
4. Skipping effect size entirely. The significance level will tell you whether there is a difference. The effect size will tell you whether it matters. You need both.
5. Ignoring a result that goes the wrong direction. If a result is statistically significant despite going in an unanticipated direction, it’s even more significant because of this very fact.

Frequently Asked Questions

What Comes Next

Do the test. Compute the effect size. Do both then take a step back and ask if the result is big enough to make any difference. But even before any of those, make sure you picked the right test based on how your data really looks — and not just because it’s easy to recall.

The t-test is easily the most commonly used statistical procedure out there. It’s also probably the most commonly misused statistical procedure. The gap between those two forms of that statement is exactly the goal of this article.