Hypothesis Testing

What Is Hypothesis Testing?

Hypothesis testing is a statistical method that uses the sample data to determine whether a claim about a population is likely true or false. The two competing hypotheses – namely the null and the alternative hypotheses – are stated, followed by the use of sample data to choose between the two.

The population represents the entire target population of interest. The sample represents the smaller portion taken from the population for analysis. Hypothesis testing exists because measuring an entire population is often impractical. Analysts test a sample instead and use probability to judge whether the result applies to the population.

For example, a firm may claim that their website gets 500 visitors per day. Using the hypothesis test, it would be determined whether that is correct by comparing the claim to the samples measured.

Two hypotheses guide all tests:

• Null Hypothesis (H₀): The initial presumption. This hypothesis suggests that there is no effect, no difference, or no relationship between variables. For example, “The mean number of daily visitors is 500.”
• Alternative Hypothesis (H₁ or Hₐ): A claim to be tested by a researcher. This hypothesis suggests that there is an effect, difference, or relationship between variables. For example, “The mean number of daily visitors is not 500.”

No test can prove a hypothesis true. Instead, it provides either sufficient or insufficient evidence to disprove the null hypothesis at a certain confidence level.

The 6 Steps of Hypothesis Testing

There is a strict sequence followed when doing a hypothesis test. Skipping a step, performing steps out of order, invalidates the result.

1. State the hypotheses. State the null hypothesis (H₀) and the alternative hypothesis (H₁) using a statistical measure in a population like the mean (μ) or the proportion (p).
2. Set a significance level (α). Determine how statistically significant your work should be before gathering data. The usual value is 0.05 (5%).
3. Select the suitable test. Select a test that is suitable to the data you are dealing with, the number of samples, and the number of groups you are comparing (refer to the table below).
4. Gather and compute. Gather data and calculate the value of the test statistics (which can either be a Z score, T score, Chi-square value, or F-value).
5. Compare the result with α. Calculate the p-value and compare it to the significance level, or compare the test statistics to the critical values in statistical tables.
6. Conclude the result. You can accept or reject the null hypothesis depending on whether the evidence presented is good enough to do so.

Choosing the Right Test: Z-Test vs. T-Test vs. Chi-Square vs. ANOVA

The correct test depends on three factors: the data type, the sample size, and the number of groups compared. Using the wrong test produces an invalid result even when every calculation is correct.

Test	Data Type	Sample Size	Groups Compared	When to Use
Z-test	Continuous (interval/ratio)	Large (n ≥ 30)	1 or 2	Population standard deviation is known, or the sample is large enough for the Central Limit Theorem to apply
T-test	Continuous (interval/ratio)	Small (n < 30) common, works for any size	1 or 2	Population standard deviation is unknown; compares one sample mean to a claim, or two group means
Chi-square test	Categorical (nominal)	Any, with expected cell counts ≥ 5	2 or more variables	Tests whether two categorical variables are independent, or whether observed counts match expected counts
ANOVA	Continuous (interval/ratio)	Any	3 or more groups	Compares means across three or more groups in a single test, avoiding repeated pairwise T-tests

Analysts choose between the Z-test and T-test based on one factor: whether the population standard deviation is known. If it is known, use the Z-test. If it is unknown and estimated from the sample, use the T-test.

Worked Examples

Z-Test Example

An organization says that its average package delivery time is 30 minutes, and the population standard deviation is 5 minutes. The mean delivery time for a sample size of 49 deliveries is 32 minutes.

Hypotheses:

• H₀: μ = 30
• H₁: μ ≠ 30

Calculation:

At α = 0.05 (two-tailed), the critical Z-value is ±1.96. The two-tailed p-value for Z = 2.80 is approximately 0.0051.

Decision: Since 0.0051 < 0.05, reject H₀. The sample provides sufficient evidence that the average delivery time differs from 30 minutes.

T-Test Example

A company claims that the mean commute time for their employees is 25 minutes. A sample of 16 employees yields a mean of 27.5 minutes with a standard deviation of 6 minutes.

Hypotheses:

• H₀: μ = 25
• H₁: μ ≠ 25

Calculation:

t = (x̄ − μ) / (s / √n) t = (27.5 − 25) / (6 / √16) t = 2.5 / 1.5 t = 1.667, with degrees of freedom (df) = n − 1 = 15

At α = 0.05 (two-tailed) with df = 15, the critical t-value is 2.131. The two-tailed p-value for t = 1.667 is approximately 0.116.

Decision: Since 0.116 > 0.05, fail to reject H₀. The sample does not provide sufficient evidence that average commute time differs from 25 minutes.

Chi-Square Test Example

A retailer wants to know whether device type (mobile or desktop) affects checkout completion. A sample of 400 sessions produces the following contingency table:

	Completed Checkout	Did Not Complete	Total
Mobile	120	80	200
Desktop	150	50	200
Total	270	130	400

Hypotheses:

• H₀: Device type and checkout completion are independent.
• H₁: Device type and checkout completion are associated.

Calculation:

Expected count for each cell = (row total × column total) / grand total. Expected counts are 135, 65, 135, and 65 for the four cells.

At α = 0.05 with df = 1, the critical chi-square value is 3.841. The p-value for χ² = 10.26 is approximately 0.0014.

Decision: Since 0.0014 < 0.05, reject H₀. Device type and checkout completion are associated in this sample.

Null vs. Alternative Hypothesis

Null and alternative hypotheses should be mutually exclusive and must cover every possible outcomes. The test also has a direction, which depends on how the alternative hypothesis is stated.

• Two-tailed test: H₁: μ ≠ μ₀. Tests the hypothesis for difference in any direction. Used where the direction of the effect is not known prior to testing.
• Right-tailed test: H₁: μ > μ₀. Tests for increase only.
• Left-tailed test: H₁: μ < μ₀. Tests for decrease only.

The two-tailed test is the default test that should be used unless there is information about direction known prior to testing. Using a one-tailed test based on data is invalid.

Significance Level, P-Value & Test Statistic

The significance level (α) refers to the predetermined probability below which the results are declared to be statistically significant. The usual significance level is 0.05, representing a 5% probability of rejecting a null hypothesis. In areas where high consequences are involved, such as in drug trials, 0.01 is used instead of 0.05.

The p-value is defined as the probability of obtaining a result as extreme as the one observed under the null hypothesis. It is important to note that the lower the p-value, the more significant the results are.

The test statistic is the standardized value derived from the data and used in making decisions on a test. Depending on the type of statistical test being performed, its formula differs: Z-score in a Z-test, t-score in a T-test, chi-square in a Chi-square test, and F-value in ANOVA.

The decision rule is consistent across all four tests:

• If p-value ≤ α, reject H₀.
• If p-value > α, fail to reject H₀.

What a P-Value Actually Means

The p-value does not give you the probability that the null hypothesis is true. Instead, the p-value gives you the probability of observing the given data, or more extreme data, under the assumption that the null hypothesis is true. This is a distinction that is frequently confused; one of the most common mistakes in statistics is mistaking p-values for probabilities that hypotheses are true.

The p-value was created by Ronald Fisher in 1925, when he published his book Statistical Methods for Research Workers. Fisher meant p-values to be one source of information used to support decision-making. The 2015 Open Science Collaboration published an article in the journal Science replicating 100 psychology papers known for statistically significant results; of those 100, only 36 produced statistically significant results on replication. The point of that article was that p-values should not be interpreted as proof of any result.

Statistically significant results (p ≤ 0.05) do not necessarily imply that the effect is big or even relevant. Large sample sizes will cause the p-value to be statistically significant for very tiny effects. Effect size will be addressed later in this section.

Type I and Type II Errors

Every hypothesis test carries two possible error types, based on the true (but unknown) state of the null hypothesis.

	H₀ Is Actually True	H₀ Is Actually False
Reject H₀	Type I Error (false positive)	Correct decision
Fail to Reject H₀	Correct decision	Type II Error (false negative)

A Type I error refers to a situation where the null hypothesis is rejected despite its truthfulness. The probability of Type I error is equal to the value of the significance level, which is represented by the symbol α.

A Type II error arises when the null hypothesis is not rejected despite its falseness. The probability of Type II error is represented by the symbol β.

Lowering α reduces the risk of a Type I error but increases the risk of a Type II error, because the likelihood of committing Type II error since rejection of H₀ becomes more difficult. By increasing the sample size, we decrease the probability of Type II error.

Statistical Power and Sample Size

Statistical power is the probability of rejecting the false null hypothesis. Power is calculated using 1 − β. In his 1988 book “Statistical Power Analysis for the Behavioral Sciences,” Jacob Cohen recommended 0.80 (80% probability of finding a true effect size) as the minimum standard of statistical power to be achieved.

Three aspects that lead to increased statistical power are:

• Larger sample size. The larger the sample size, the lower the sampling error and hence the easier it is to detect the true effects.
• Larger effect size. The larger the true differences between groups, the easier it is to detect the difference than when the difference is smaller.
• High significance level (α). The less restrictive the significance level (α), the higher the power, but it increases Type I errors. There is a tradeoff relationship between the two.

The sample size calculator uses three variables: The significance level, the power (usually set at 0.80) and the effect size.

Effect Size

An effect size is the standardized amount of difference or association between two variables regardless of sample size. While statistical significance tells us whether there is an effect, effect size tells us the magnitude of that effect.

The most frequently used effect size for a comparison of the means of two groups is Cohen’s d:

d = (mean of Group A − mean of Group B) / pooled standard deviation

For example, Group A averages 82 points, Group B averages 78 points, and the pooled standard deviation is 8 points.

Cohen (1988) defined d = 0.2 as a small effect, d = 0.5 as a medium effect, and d = 0.8 as a large effect. It is possible that a statistically significant finding with a small effect size (d < 0.2) is of little practical significance, even if p < 0.05.

A Short History: Fisher vs. Neyman-Pearson

Two different statistical approaches underpin the hypothesis testing as practiced today and were devised separately from each other.

Ronald Fisher pioneered the idea of significance testing in the 1920s where the core idea was based around the notion of p-value which was viewed as a continuously varying measure of evidence against a specific null hypothesis. The Fisher approach does not involve setting an alternative hypothesis or a significance level beforehand.

Another approach that was devised by Jerzy Neyman and Egon Pearson in 1933 is based on the idea of two mutually exclusive hypotheses (H₀ and H₁) along with the idea of a predetermined significance level as well as Type I and Type II errors. The Neyman-Pearson lemma was formulated in the same time frame.

The modern test of hypotheses includes components from both theories: p-values and significance levels from Fisher, and the two-hypotheses, fixed-error-rates framework from Neyman and Pearson. It has been noted by statisticians that this mixed heritage explains many misunderstandings regarding the meaning of p-values.

Where Hypothesis Testing Shows Up in Real Life

• Business: In A/B Testing, two variants of a website, application feature, or marketing strategy are compared in order to see if one variant significantly improves the target metric.
• Healthcare: In clinical trials, hypothesis testing is used to compare a treatment group with a control group and find out whether the impact of a drug is better than that by chance.
• Economics: Economic analysts apply hypothesis testing to find out whether the introduction of new policies, changes in prices, or any market events have caused significant changes to occur in the economy.
• Manufacturing: Hypothesis testing is applied in quality control in order to find out whether the batch under analysis differs from a given tolerance.

Is Hypothesis Testing Still Relevant With AI and Automated Testing?

Yes, machine learning systems rely on hypothesis testing to prove their efficacy, features and performance as well as A/B test results. There are automated platforms that perform hundreds of hypothesis tests in a day. However, the logic behind it all – that is, null and alternative hypotheses, p-values and types I and II errors – have not changed. In addition, data scientists use hypothesis testing to verify the performance of new models against existing ones.

Common Mistakes to Avoid

1. P-hacking. Testing for significance in a set of data repeatedly until p < 0.05 makes the true Type I error rate higher than stated in α.
2. Neglecting the magnitude of the difference between means. A statistically significant difference may not warrant business or clinical action because of a small effect size.
3. Using the wrong statistical test. Applying the T-test for categorical variables, or using the Z-test when the standard deviation is unknown, results in an invalid test.
4. Interpreting “cannot reject H₀” as evidence that H₀ is true. A non-significant result may mean only that the test lacks the ability to detect a true effect.
5. Selecting a one-tailed test based on observed data. Directionality for a one-tailed test must be decided beforehand.

Frequently Asked Questions