Linear Regression has 6 assumptions, not 4 or 5. The number varies in various articles due to the merging or omission of two assumptions, namely the independent error terms and endogeneity. This article covers all 6 assumptions and applied each of these assumptions on one sample and showed how to correct the violation.
In the sample data set used in this article, the predictor variable is hours studied and the response variable is exam score based on 40 observations. The same sample data set is used throughout all the assumptions mentioned below.
What Will I Learn?
Why These Assumptions Exist
Linear Regression estimates the coefficients through the use of the Ordinary Least Squares (OLS) method. The OLS algorithm determines the line that yields the smallest sum of squared errors.
The Gauss-Markov theorem indicates that OLS results in the Best Linear Unbiased Estimator (BLUE) under certain conditions. “Best” refers to having the smallest variance compared to other linear unbiased estimators. If any one of these conditions does not hold, then the OLS algorithm will run and yield coefficients but they are no longer statistically valid. Confidence intervals cannot be trusted anymore. P-values are no longer valid either.
Each assumption listed below corresponds to a condition of the Gauss-Markov theorem.
Machine Learning Course
Average time: 5 month(s)
Skills you’ll build: Python, Scikit-learn, Supervised & Unsupervised Learning, Feature Engineering, Model Deployment, and more..
1. Linearity
Linearity means the relationship between the independent variable and the dependent variable follows a straight line. Any increase by one unit in the independent variable must result in an increase in the same amount of change in the dependent variable, irrespective of where that unit increase is made on the scale.
With regard to the exam-score illustration, linearity ensures that any extra hour of studying yields a similar increase in the score, whether the student has spent 2 hours or 12 hours studying.
How to Test Linearity
Draw a scatterplot of the residual versus the fitted values. Scattered data points without any curve pattern indicate linearity. On the other hand, if a curvilinear trend can be seen, such as data points that increase, decrease, and increase again, then there is evidence of a non-linear relation.
The Ramsey regression equation specification error test (RESET) is a more rigorous approach. It involves adding power functions of the fitted values to the regression. If this is statistically significant (at p < 0.05), then the linear equation is mis-specified.
How to Fix a Linearity Violation
| Situation | Fix |
|---|---|
| Relationship curves upward or downward | Add a squared or cubic term for the independent variable |
| Relationship grows exponentially | Apply a logarithmic transformation to the dependent variable |
| Relationship changes direction | Switch to a non-linear model (polynomial regression, splines) |
2. Independence of Errors
Error independence is that the errors should not be correlated with each other. The prediction errors must be independent of each other.
The error independence condition holds for the time series data. In the example of exam scores, then the 40 students must be tested independently, with no shared conditions (such as the same exam session or seating arrangement) linking their errors together.
How to Test Independence
The Durbin-Watson test measures autocorrelation in residuals. The test statistic ranges from 0 to 4.
| Durbin-Watson value | Interpretation |
|---|---|
| Close to 2 (typically 1.5–2.5) | No autocorrelation |
| Below 1.5 | Positive autocorrelation |
| Above 2.5 | Negative autocorrelation |
How to Fix an Independence Violation
Include the time-lagged independent variable in the regression equation, use a time-series method (e.g., ARIMA, GLS with autocorrelated errors), or gather the data with an experimental design eliminating the common element that creates the correlation.
3. Homoscedasticity
If the data has homoscedasticity, it means that the variance in the residuals does not change regardless of the values of the independent variable. The variance in prediction errors should be stable whether the independent variable takes on small values or large values.
Heteroscedasticity is an alternative to homoscedasticity, where there is some kind of systematic change in the spread of residuals. In the case of the exam score prediction example, heteroscedasticity will exist when predictions for low hours of study are accurate but predictions for high hours of study vary significantly.
How to Test Homoscedasticity
Plot the residuals versus fitted values. A pattern of constant-width horizontal band shows that there is no problem of heteroscedasticity. On the other hand, the pattern where the plot of residuals takes funnel shape is the evidence of heteroscedasticity.
For testing heteroscedasticity, there are two tests which can be used, namely, Breusch-Pagan test and Goldfeld-Quandt test. For the Breusch-Pagan test, the squared residuals are regressed against the independent variables, while for the Goldfeld-Quandt test, data is divided into two parts and variance of residuals for each group is compared. Both these tests give the value of p. If p < 0.05, then there is a heteroscedasticity problem.
How to Fix a Homoscedasticity Violation
Take the log or square root of the dependent variable, or employ Weighted Least Squares (WLS) rather than OLS. In WLS, less weight is assigned to the data points with larger residual variance.
4. Normality of Residuals
A normally distributed pattern of residuals implies that the errors in prediction will have a normal distribution with zero mean. The majority of the residuals will be clustered around zero, with fewer residuals at increasing distances from zero.
The normality assumption is more relevant to statistical inference than to prediction per se. This is because even when the normality assumption does not hold, a regression model can make reasonable point predictions.
How to Test Normality
The Q-Q (Quantile-Quantile) plot is the graphical representation of residuals versus the expected normal distribution. Points lying on the diagonal straight line show normality. Departure from the line at either end implies the presence of skewness and heavy tails.
The Shapiro-Wilk test is used as a statistical method to check normality when the number of samples is less than 5,000. A p-value of 0.05 shows deviation from normality.
How to Fix a Normality Violation
Transform the dependent variable using either a log, a square root, or a Box-Cox transformation. Consider removing or examining any extreme outliers, as a few extreme residuals are usually what cause the deviation from normality. Larger sample sizes are helpful in cases like this due to the Central Limit Theorem.
5. No Multicollinearity
In case of no multicollinearity, the independent variables are not highly correlated. Each independent variable needs to bring something new to the equation.
Multicollinearity does not bias predictions, but it inflates the variance of coefficient estimates. This results in unstable and ambiguous coefficients. Suppose there is a regression analysis to predict test scores based on study hours and review hours; in such a case, the high correlation between both variables will make it hard to isolate their respective impacts.
How to Test Multicollinearity
The Variance Inflation Factor (VIF) determines the extent to which the variance of a particular coefficient is increased due to collinearity with another independent variable. The VIF value is determined independently for each individual independent variable.
| VIF value | Interpretation |
|---|---|
| 1 | No correlation with other predictors |
| 1–5 | Moderate correlation, generally acceptable |
| 5–10 | High correlation, investigate further |
| Above 10 | Severe multicollinearity |
How to Fix a Multicollinearity Violation
Exclude one of the correlated variables from the model. Create an index of correlated variables to reduce them into one variable. Use either Ridge regression or Lasso regression, both of which discourage large coefficients in order to lessen the effects of multicollinearity.
Machine Learning Course
Average time: 5 month(s)
Skills you’ll build: Python, Scikit-learn, Supervised & Unsupervised Learning, Feature Engineering, Model Deployment, and more..
6. No Endogeneity
There is no endogeneity if the explanatory variables and the error term are uncorrelated. It is the assumption that is usually left out in the shorter list, as well as the one that is difficult to check just by examining the residuals.
Endogeneity arises in three main ways: omitted variable, which has effects on the explanatory variable and the dependent variable; measurement error of the explanatory variable; and reverse causality, whereby the dependent variable has effects on the explanatory variable. In the example of exam scores, endogeneity will be caused by the student’s motivation, which is an omitted variable.
How to Test for Endogeneity
Endogeneity cannot be verified by way of a single test carried out on the basis of the data set. Endogeneity should be verified using the understanding of the data generation process in determining the presence of the existence of omitted variables, measurement errors, and reverse causality. The Hausman Test compares OLS estimates against an instrumental-variables estimate; a statistically significant difference between the two indicates endogeneity is present.
How to Fix an Endogeneity Violation
Use instrumental variables (IV) regression. The instrumental variable is related to the endogenous independent variable but has no direct influence on the dependent variable, thus isolating the variation that is required for estimation. By adding the omitted variable into the model, if it is observable, the source of endogeneity is immediately addressed.
Testing All 6 Assumptions in Python
The following code tests all 6 assumptions using the exam-score dataset and the statsmodels library.
import numpy as np
import pandas as pd
import statsmodels.api as sm
from statsmodels.stats.diagnostic import het_breuschpagan
from statsmodels.stats.stattools import durbin_watson
from statsmodels.stats.outliers_influence import variance_inflation_factor
from scipy import stats
# Fit the model
X = sm.add_constant(df[['hours_studied']])
y = df['exam_score']
model = sm.OLS(y, X).fit()
residuals = model.resid
fitted = model.fittedvalues
# 1. Linearity -- Ramsey RESET test
from statsmodels.stats.diagnostic import linear_reset
reset_result = linear_reset(model, power=2, use_f=True)
print(f"RESET test p-value: {reset_result.pvalue:.4f}")
# 2. Independence -- Durbin-Watson
dw_stat = durbin_watson(residuals)
print(f"Durbin-Watson statistic: {dw_stat:.4f}")
# 3. Homoscedasticity -- Breusch-Pagan
bp_test = het_breuschpagan(residuals, X)
print(f"Breusch-Pagan p-value: {bp_test[1]:.4f}")
# 4. Normality -- Shapiro-Wilk
shapiro_stat, shapiro_p = stats.shapiro(residuals)
print(f"Shapiro-Wilk p-value: {shapiro_p:.4f}")
# 5. Multicollinearity -- VIF (requires 2+ predictors)
vif_data = pd.DataFrame()
vif_data["variable"] = X.columns
vif_data["VIF"] = [variance_inflation_factor(X.values, i) for i in range(X.shape[1])]
print(vif_data)
# 6. Endogeneity -- requires domain review; no single test confirms it
print(model.summary())P-values of more than 0.05 on the RESET test, Breusch-Pagan test, and Shapiro-Wilk test justify the assumption. The Durbin-Watson statistic within the range of 1.5 to 2.5 justifies the assumption of independence. A VIF below 5 supports the absence of multicollinearity.
Summary Table: Assumption, Test, and Fix
| # | Assumption | How to Test | How to Fix |
|---|---|---|---|
| 1 | Linearity | Residual vs. fitted plot; Ramsey RESET test | Add polynomial terms; log-transform the dependent variable |
| 2 | Independence of Errors | Durbin-Watson test | Add lagged variables; switch to a time-series model |
| 3 | Homoscedasticity | Residual vs. fitted plot; Breusch-Pagan test | Log/square-root transformation; Weighted Least Squares |
| 4 | Normality of Residuals | Q-Q plot; Shapiro-Wilk test | Box-Cox transformation; remove extreme outliers |
| 5 | No Multicollinearity | Variance Inflation Factor (VIF) | Remove or combine variables; Ridge or Lasso regression |
| 6 | No Endogeneity | Hausman test; domain review | Instrumental variables regression; add omitted variable |
What to Do When an Assumption Is Violated
Violation of one assumption does not necessarily mean that the regression analysis cannot be performed. This depends on the violated assumption and the purpose of the analysis.
If the model is used only for prediction purposes, and there is no violation of the assumptions regarding linearity and homoscedasticity, the assumption of normality does not really matter. Point predictions can still be utilized even if confidence intervals are not valid.
If the model is used for inference to test the statistical significance of the coefficient, then a violation of normality, homoscedasticity, or independence violates the validity of p-values and confidence intervals. Make necessary corrections to your model.
If there is multicollinearity but the model is used for prediction, this violation of an assumption can be safely ignored because multicollinearity affects interpretation of individual coefficients more than prediction accuracy.
If there is endogeneity, the coefficients of the model will be biased regardless of the model’s purpose. This violation needs to be addressed before any output can be trusted.
Frequently Asked Questions
Q1. How many assumptions does linear regression have?
Ans. There are 6 assumptions in linear regression: linearity, independence of error terms, homoscedasticity, normality of residuals, absence of multicollinearity and absence of endogeneity.
Q2. Do these assumptions apply to simple linear regression, multiple regression, or both?
Ans. All six assumptions apply both to simple linear regression (where there is only one independent variable) and to multiple linear regression (where there are at least two independent variables). The assumption of multicollinearity applies to regression where there are at least two independent variables.
Q3. What is the most commonly violated assumption in real-world data?
Ans. The assumptions of homoscedasticity and normality of residuals are the assumptions that are violated most often in practice, since in practice use variable variance and skewed distribution of residuals.
Q4. Can a regression model still be trusted if one assumption is only slightly violated?
Ans. A minor violation of normality or homoscedasticity generally has a limited effect on regression results, particularly with larger sample sizes.However, the violation of the assumption of endogeneity will always bias the model.
Q5. What is the difference between homoscedasticity and multicollinearity?
Ans. Homoscedasticity is all about the variability of the error terms while multicollinearity relates to the interrelationship between the independent variables.
Conclusion
This particular exam score model was verified against all 6 assumptions and found to satisfy linearity, independence, homoscedasticity, and normality. The VIF test showed that there is no multicollinearity problem as only one independent variable was involved. The endogeneity assumption demanded an evaluation of how the data was collected, and no possibility of omitted variable was found for hours studied.
Perform the 6-test run of any regression model prior to presenting the coefficients. When the model satisfies all 6 criteria, the regression coefficient estimates will satisfy the criteria of the Gauss-Markov theorem and hence are statistically valid.
