Linear Regression in Machine Learning

When that “28 minutes” notification pops up for every food delivery service app you’ve ever used, that prediction was made using linear regression. The machine learning model takes all the historical data about distances, time required for order preparation, and other factors that could influence delivery time and finds the most accurate line to predict delivery time from these inputs. There you have it — that’s all there is to it!

Linear regression is the simplest form of a supervised machine learning algorithm that predicts the value of an outcome variable as a function of one or more independent variables using linear relationships. And it happens to be the oldest algorithm in machine learning — so old, in fact, that in 2026 some of the biggest corporations around the globe are still using linear regression models in production due to their efficiency and interpretability.

However, while many tutorials will teach you how to use linear regression, only a few will explain when you should be using it. This guide does both!

What Is Linear Regression?

Linear Regression

Linear regression is an instance of a supervised learning model that learns how to predict the value of a numeric quantity based on one or several numeric features by fitting the optimal straight line to the training examples.

The notion of “regression” itself has a fascinating story to be told. In the 1880s, statistician Francis Galton examined correlations between the heights of parents and their kids and coined the phrase “regression towards mediocrity” because unusually tall parents generally had tall but somewhat shorter-than-them children. The statistical technique he invented to characterize this phenomenon is linear regression.

Linear regression, also known as Ordinary Least Squares (OLS) regression, is the standard kind, and when you say just linear regression without any additional clarification, you are most likely referring to OLS regression.

The basic equation looks like this: 

Where:

• ŷ = the predicted output (your estimate of Y)
• β₀ = the intercept (the value of Y when all X inputs equal zero)
• β₁, β₂…βₙ = coefficients (how much Y changes per one-unit increase in each X)
• x₁, x₂…xₙ = your input features

The Greek letters should not scare you, as these are nothing but numbers generated by the algorithm using the data. The whole objective of the algorithm is to identify β₀, β₁, etc., such that the error in prediction is minimized.

How Linear Regression Actually Works

The Best-Fit Line: What the Algorithm Is Trying to Do

Think of how hard it would be to plot a straight line on a scatter graph that would pass through all the data points. The idea is to minimize the distance between each data point and the straight line that we plotted. This is exactly what the algorithm is set out to do, but in mathematical terms.

The distance between the value predicted by the model and the actual value is referred to as a residual. If the residual value is positive, then there is an under-prediction problem.

Residuals and the Least Squares Method

Linear regression employs Least Square Methods to determine the optimal line since this method minimizes the sum of squares of the residuals, and not the residuals themselves.

Why squares of residuals and not residuals? There are two reasons. Firstly, squaring ensures that the residuals are always positive (negative residual will offset positive residual). Secondly, it penalizes higher residuals fourfold as compared to smaller ones (10 unit error is four times worse than 5 units).

The Residual Sum of Squares Formula is:

Where yᵢ is the actual observed value and ŷᵢ is what the model predicted for that row.

The line that minimizes RSS is the best-fit line. Full stop.

The Cost Function (Mean Squared Error)

The cost function is how the algorithm measures how wrong it is during training. For linear regression, the standard cost function is Mean Squared Error (MSE) — the average squared residual across all training examples:

Lower MSE means a better-fitting model. The algorithm’s job is to keep adjusting the coefficients until MSE is as low as possible.

Gradient Descent: Finding the Best Coefficients Iteratively

This is how gradient descent works in practice. Imagine that you find yourself in a hill-covered area with a blindfold on your eyes, with a goal to find the bottom of the deepest valley there. You cannot see anything in front of you, but you can tell where the ground slopes down from under your feet. Therefore, you walk that direction until you find the bottom.

As simple as that! Gradient descent starts with randomly generated coefficient values, calculates the derivative of the function representing the loss curve (called “gradient”) and shifts the coefficients accordingly, making the process of minimization repeat until no more progress is being made.

The magnitude of each movement is controlled by the learning rate (α), which is probably the most difficult hyperparameter to tune properly:

• Too Large: the algorithm will overstep the minimum and start oscillating around it indefinitely
• Too Small: the algorithm will be walking for too long and converge much slower than necessary
• Good Starting point: usually it’s fine to begin with α = 0.01

[VISUAL: Loss surface diagram — 3D bowl shape with a ball rolling downhill via gradient descent steps, annotated with “learning rate too large = overshooting” and “good learning rate = converging” — gradient descent linear regression visualization]

The Normal Equation: The Closed-Form Alternative

However, gradient descent is an iterative algorithm; it requires multiple steps until convergence. However, there is a second way of solving linear regression problems which tutorials rarely mention the Normal Equation method.

The Normal Equation directly computes the optimal parameters in one go:

This gives you the exact optimal coefficients. No learning rate to tune. No convergence to wait for.

So when should you use it instead of gradient descent?

Decision	Use Normal Equation	Use Gradient Descent
Dataset size	< ~10,000 rows	> 10,000 rows
Number of features	< ~1,000 features	> 1,000 features
Need exact solution	Yes	Less critical
Training speed matters	Less critical	Yes

The catch: inverting the matrix (XᵀX)⁻¹ is computationally expensive. For large datasets with many features, it becomes prohibitively slow. That’s why scikit-learn uses gradient descent by default at scale.

Simple vs Multiple Linear Regression

Simple Linear Regression: One Predictor

Simple linear regression uses exactly one input feature to predict the output. The equation reduces to:

Let’s do this with real numbers. Suppose you have five students and you want to predict their exam scores from hours studied:

Hours Studied (x)	Exam Score (y)
1	50
2	55
3	65
4	70
5	80

Step 1: Calculate the means

• x̄ = (1+2+3+4+5)/5 = 3
• ȳ = (50+55+65+70+80)/5 = 64

Step 2: Calculate the slope (β₁)

β₁ = Σ[(xᵢ – x̄)(yᵢ – ȳ)] / Σ[(xᵢ – x̄)²] Numerator: (1-3)(50-64) + (2-3)(55-64) + (3-3)(65-64) + (4-3)(70-64) + (5-3)(80-64)         = (-2)(-14) + (-1)(-9) + (0)(1) + (1)(6) + (2)(16)         = 28 + 9 + 0 + 6 + 32 = 75 Denominator: (-2)² + (-1)² + 0² + 1² + 2² = 4 + 1 + 0 + 1 + 4 = 10 β₁ = 75 / 10 = 7.5

Step 3: Calculate the intercept (β₀)

Final model: ŷ = 41.5 + 7.5x

This means that for each extra hour of study, the model predicts an increase of 7.5 points in the test score. Try this out using scikit-learn, and your results should be β₀ ≈ 41.5 and β₁ ≈ 7.5.

The vast majority of tutorial material simply skips this step. It takes the shortcut through scikit-learn without ever showing you that scikit-learn is just doing this math much faster than you can.

Multiple Linear Regression: Many Predictors

Multiple linear regression extends the formula to handle more than one input feature:

Suppose that you are now forecasting test scores based on both hours spent studying (x₁) and hours slept prior to the test (x₂). The meaning of each parameter is clear: “while holding everything else constant, a one-unit rise in x₁ results in β₁ unit change in ŷ.”

The term “everything else held constant” is the critical one here, and that’s why a separate parameter can be interpreted despite using lots of features.

Dummy variables for categorical features

What if your independent variable is actually categorical, such as “tutoring: yes or no”? Linear regression requires numbers. So you create a dummy variable where 0 = no tutoring and 1 = tutoring. Your model would include variables such as:

The coefficient 8.4 means: students who received tutoring scored 8.4 points higher on average, after accounting for study hours and sleep. That’s directly actionable information.

Watch out for multicollinearity

Where two independent variables show very high correlation, for instance, “TV Hours” and “Sleep Hours” both indicating leisure time, the effect of each becomes hard to isolate. This phenomenon is called multicollinearity and causes problems in determining the impact of individual variables, although it need not affect the performance of the entire model negatively. The usual measure used for this purpose is the Variance Inflation Factor (VIF), which should be below 10. [INTERNAL LINK: multicollinearity deep dive]

The 6 Assumptions of Linear Regression (And How to Test Each One)

Every competitor lists these assumptions. None of them tell you how to actually check them. Here’s the practical version.

Assumption	Plain English	How to Diagnose Violations
Linearity	The relationship between X and Y follows a straight line	Plot residuals vs. fitted values — look for curves or patterns
Independence of errors	Each prediction error is unrelated to the others	Durbin-Watson test; critical in time-series data
Homoscedasticity	Errors have equal spread across all values of X	Residual plot — spread should look like a random horizontal band, not a funnel
Normality of errors	Prediction errors follow a bell-curve distribution	QQ plot of residuals — points should fall near the diagonal line
No multicollinearity	Input features aren’t too closely correlated with each other	Variance Inflation Factor (VIF) — flag anything above 10
No autocorrelation	Errors don’t follow a repeating pattern over time	Durbin-Watson statistic — values near 2.0 suggest no autocorrelation

What to do when an assumption fails

• Linearity Violated: try using polynomial regression or applying the logarithm to Y 
• Heteroscedasticity found: use weighted least squares or apply a logarithm to Y 
• Non- normal residuals: first examine for outliers; large samples tend to self-correct through the Central Limit Theorem 
• Multicollinearity detected: eliminate one of the correlated variables or go with Ridge Regression 
• Autocorrelation present : most likely requires time series analysis rather than OLS

Evaluating Your Model: Metrics That Actually Matter

R-Squared: The Goodness-of-Fit Score

The R-square indicates how much of the variation in Y is accounted for by your model. With an R-Square value of 0.85 in your model, your model accounts for 85% of the variation in the dependent variable, while the other 15% is just noise.

Where TSS is the Total Sum of Squares — the variance you’d have if you just predicted the mean of Y every time.

R-squared runs from 0 to 1 in well-fitted models. A negative R² is possible — it means your model is doing worse than simply predicting the average. That’s a serious problem worth investigating.

But don’t over-trust R-squared. A high R-squared doesn’t mean good predictions on new data. Add five useless random features to your model and R-squared goes up — even though you’ve learned nothing useful. That’s where adjusted R-squared comes in.

Adjusted R-Squared: The Honest Version for Multiple Regression

Adjusted R-squared penalizes you for adding predictors that don’t genuinely improve the model:

Where n is the number of observations and k is the number of predictors.

Rule of thumb: always report adjusted R-squared when your model has more than one predictor. If adding a new feature increases R² but decreases adjusted R², that feature is probably noise.

RMSE and MAE: Error in the Units That Actually Matter

Root Mean Squared Error (RMSE) gives you prediction error in the same units as your target variable. If you’re predicting house prices in dollars, RMSE of $15,000 means your predictions are off by roughly $15,000 on average. That’s interpretable in a way that MSE (which would be $225,000,000) is not.

Mean Absolute Error (MAE) is simpler: just the average absolute difference between predictions and actual values.

When should you use each?

Situation	Best Metric
Outliers present in your target	MAE — it doesn’t square errors, so outliers don’t dominate
You want to penalize large misses heavily	RMSE — it punishes big errors more
Reporting results to non-technical stakeholders	RMSE (same units as target, easier to explain)
All errors equally important	MAE

[VISUAL: Metric decision table as a clean styled table — linear regression evaluation metrics comparison]

Interpreting Coefficients in Plain English

This is the most practical skill in all of regression analysis, and it gets almost no attention.

Each coefficient tells you the expected change in Y for a one-unit increase in that predictor, holding everything else constant. Here are three domain-specific examples to make that concrete:

Example 1 — Real estate Model: Price = 50,000 + 120(sqft) + 8,000(bedrooms)

The coefficient 120 means: each additional square foot adds $120 to the predicted price. The coefficient 8,000 means: each additional bedroom adds $8,000, holding square footage fixed. The intercept $50,000 is the predicted price when sqft = 0 and bedrooms = 0 — which is meaningless in practice. You often ignore the intercept unless zero is a plausible value for all inputs.

Example 2 — Marketing Model: Revenue = 5,000 + 4.2(ad_spend)

Every dollar spent on advertising generates $4.20 in predicted revenue. If that coefficient is statistically reliable, it directly informs your budget decisions.

Example 3 — Education Model: Score = 41.5 + 7.5(hours_studied)

We built this model earlier from scratch. The slope 7.5 says: one more hour of studying predicts 7.5 more points on the exam. That’s something a student can act on.

The intercept (41.5) means a student who studied zero hours is predicted to score 41.5. Whether that makes real-world sense depends on context — but the coefficient 7.5 is the actionable insight.

One thing I think most regression tutorials get wrong: they present coefficients as mere numbers to report, not as business insights to act on. The coefficient IS the finding. Everything else — the R-squared, the p-values, the model accuracy — is just quality assurance for that number.

When NOT to Use Linear Regression

This section doesn’t exist anywhere in the top-ranking results. That’s a gap worth filling, because misapplying linear regression is one of the most common mistakes in applied machine learning.

Here’s the decision framework:

Stop. Don’t use linear regression if…

1. Your output variable is categorical (not a number) If you’re predicting “will this customer churn: yes or no,” that’s a classification problem. Linear regression can technically output a number between 0 and 1 here, but it has no mechanism to constrain predictions to that range — you’ll get predictions like -0.3 or 1.4. Use logistic regression instead.

2. The relationship between X and Y is clearly non-linear. Plot your data first. If the pattern curves — exponential growth, a U-shape, a plateau — a straight line will fit badly regardless of how much data you have. Try polynomial regression or a tree-based model.

3. Severe outliers exist in your dataset Linear regression is not resistant to outliers. One extremely aberrant data point can drag the entire fitted line off course. First try to understand why that outlier exists. If it’s a data entry error, remove it. If it’s real, consider robust regression methods or median-based approaches.

4. Your target variable is count data (0, 1, 2, 3…) Predicting the number of customer support tickets filed per day is a count problem. Linear regression can predict negative counts, which is nonsensical. Use Poisson regression.

5. Your input features are highly correlated with each other. Multicollinearity makes individual coefficients unreliable and hard to interpret. Switch to Ridge regression (L2 regularization), which handles correlated features gracefully.

6. You’re working with time-series data that has autocorrelation. If your data is ordered over time and past errors predict future errors, standard OLS violates the independence assumption in a way that can seriously distort your results. Look at ARIMA, SARIMA, or time-series–specific models.

7. You have far more features than observations (p >> n) With 500 features and only 200 rows, OLS will overfit badly and become numerically unstable. Use Lasso regression — it actively shrinks irrelevant coefficients to zero, performing feature selection as part of fitting.

Linear Regression vs Other Algorithms

When linear regression isn’t the right tool, here’s what is. This comparison covers the most common alternatives you’ll encounter in practice.

Algorithm	Best For	Handles Non-Linearity?	Interpretability	Switch From LR When…
Linear Regression (OLS)	Linear relationships, continuous output	No	Excellent — coefficients have direct meaning	—
Ridge Regression (L2)	Multicollinearity, all features matter	No	Good — coefficients still interpretable	Correlated features inflate coefficient variance
Lasso Regression (L1)	High-dimensional data, built-in feature selection	No	Good — zeros out irrelevant coefficients	You have many features and suspect most are noise
Polynomial Regression	Curved relationships	Yes (up to degree N)	Moderate — coefficients get abstract	Residual plot shows a clear curve pattern
Decision Tree Regressor	Non-linear relationships, feature interactions	Yes	High — visual and intuitive	Data has complex interactions between features
Gradient Boosting (XGBoost, LightGBM)	Complex non-linear tabular data	Yes	Low — black-box	Accuracy is priority and interpretability is secondary

One useful rule of thumb: start with linear regression. If the model fits well and meets your accuracy needs, stop there. Complexity should be earned, not assumed.

[INTERNAL LINK: regularization methods — Ridge vs Lasso vs Elastic Net]

Regularization: When Standard Linear Regression Overfits

Ridge Regression (L2)

Ridge regression adds a penalty term to the cost function that shrinks all coefficients toward zero — but never all the way to zero. It’s the right choice when you believe all your features contribute something, but some coefficients are being inflated by multicollinearity or noise.

The Ridge cost function:

The λ (lambda) controls how much to shrink. Higher λ = more shrinkage = simpler model.

Lasso Regression (L1)

Lasso uses absolute values instead of squares in its penalty term. The critical difference: Lasso can shrink coefficients all the way to exactly zero, effectively removing those features from the model. It does feature selection automatically.

The Lasso cost function:

Use Lasso when you suspect only a handful of your many predictors actually matter.

Elastic Net

Elastic Net combines both penalties — L1 for sparsity and L2 for stability with correlated features. It’s the safest default when you’re unsure which regularization to use and you have correlated predictors.

Python Implementation: From Data to Predictions

Let’s implement the exam score example from earlier. Same data, same math — now in code.

Simple Linear Regression with scikit-learn

python
import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error, r2_score
from sklearn.model_selection import train_test_split

# Our 5-row dataset from the worked example above
X = np.array([[1], [2], [3], [4], [5]])   # hours studied
y = np.array([50, 55, 65, 70, 80])         # exam scores

# Split into train/test (for real projects -- here we'll fit on all data for demonstration)
# X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Fit the model
model = LinearRegression()
model.fit(X, y)

# Check our coefficients -- should match β₀=41.5, β₁=7.5 from the manual calculation
print(f"Intercept (β₀): {model.intercept_:.2f}")    # → 41.50
print(f"Slope (β₁): {model.coef_[0]:.2f}")           # → 7.50

# Make a prediction: how many points for 6 hours of study?
prediction = model.predict([[6]])
print(f"Predicted score for 6 hours: {prediction[0]:.1f}")  # → 86.5

# Evaluate
y_pred = model.predict(X)
print(f"R-squared: {r2_score(y, y_pred):.3f}")
print(f"RMSE: {np.sqrt(mean_squared_error(y, y_pred)):.3f}")

The output should confirm β₀ ≈ 41.5 and β₁ ≈ 7.5 — exactly what we calculated by hand. If you’re seeing different numbers, check for a data entry issue.

Multiple Linear Regression with scikit-learn

python
import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression
from sklearn.metrics import r2_score
from sklearn.model_selection import train_test_split

# Extending to 3 features: hours studied, hours slept, tutoring (0/1)
data = {
    'hours_studied': [1, 2, 3, 4, 5, 2, 3, 4, 5, 6],
    'hours_slept':   [6, 7, 6, 8, 7, 5, 8, 7, 9, 8],
    'tutoring':      [0, 0, 0, 0, 0, 1, 1, 1, 1, 1],   # dummy variable
    'score':         [50, 55, 65, 70, 80, 62, 75, 82, 90, 95]
}
df = pd.DataFrame(data)

X = df[['hours_studied', 'hours_slept', 'tutoring']]
y = df['score']

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=42)

model = LinearRegression()
model.fit(X_train, y_train)

# Print coefficients with feature names
for feature, coef in zip(X.columns, model.coef_):
    print(f"{feature}: {coef:.2f}")
print(f"Intercept: {model.intercept_:.2f}")

# Test set performance
y_pred = model.predict(X_test)
print(f"\nTest R-squared: {r2_score(y_test, y_pred):.3f}")

Diagnosing the Model with Residual Plots

python
import matplotlib.pyplot as plt

# After fitting any LinearRegression model:
y_pred_train = model.predict(X_train)
residuals = y_train.values - y_pred_train

plt.figure(figsize=(8, 4))
plt.scatter(y_pred_train, residuals, alpha=0.7)
plt.axhline(y=0, color='red', linestyle='--', linewidth=1)
plt.xlabel('Fitted Values')
plt.ylabel('Residuals')
plt.title('Residual Plot -- check for patterns')
plt.tight_layout()
plt.show()

What you’re looking for: a cloud of points randomly scattered around the zero line. Funnel shaped pattern (the scatter increases with the fitted values), then there is heteroscedasticity (non-constant error variance). The pattern of curve suggests violation of the assumption of linearity. In either case, there is a need for model modification.

Real-World Applications of Linear Regression in 2026

Linear regression is not a “beginner’s algorithm you graduate from.” It’s in active production use across industries because it’s fast, explainable, and good enough for a surprisingly wide range of problems.

Energy consumption forecasting for data centers The major cloud providers employ linear regression to estimate hourly power consumption using server utilization rate, room temperature, and time of day variables. Retraining takes place each day, and speed of retraining in seconds for OLS versus several hours for neural nets keeps LR in the running.

SaaS churn probability proxies Whereas logistic regression is employed in the actual prediction process due to the binary nature of the output, product teams employ linear regression to estimate “days until next login” or “monthly active usage hours” (both continuous), which in turn have a relationship with the likelihood of churn.

Real estate price estimation Zillow, Redfin, and similar platforms run ensemble models, but linear regression components remain in the stack because their coefficients are legally defensible in fair housing audits. An algorithm that says “brick exterior adds $8,200 to value” can be explained to a regulator. A gradient boosting model with 500 trees cannot. 

Credit risk scoring Banks still use OLS variants in regulated credit models because regulators require explainability. The Equal Credit Opportunity Act in the US requires that lenders be able to provide specific, articulable reasons for credit decisions — something black-box models struggle to satisfy. 

Marketing mix modeling Media planners rely on linear regression for the apportionment of revenue to various marketing channels. These include television, digital, search, and social media. The value attached to each variable is the revenue made per dollar spent on the particular channel.

LLM compute cost estimation Emerging use case (2026): Engineering teams working at AI firms apply linear regression for predicting the inference cost using input tokens, model size, and hardware generation. In this case, the association between the variables is truly linear on the log scale, and the ease of applying the model contributes significantly to its success.

Frequently Asked Questions

Conclusion

It is worth investing your time into comprehending the fundamentals of linear regression because although the method may be considered straightforward, all ideas underlying the cost functions, gradient descent, coefficients and residual diagnostics, will be utilized throughout almost any advanced machine learning technique.

To begin, start working with the model. Estimate your coefficients from existing data. Try reading them aloud like “for each extra hour spent studying, we predict 7.5 additional points on the exam.” Is this assumption reasonable? Now, conduct the residual diagnostics procedure and evaluate your assumptions. This process performed honestly is capable of giving more insights into the data structure than any further analysis.

Algorithms that emerge afterwards can only become more powerful if linear regression is not sufficient anymore.