Regression Explained

You carried out a regression test; you have obtained R² equal to 0.99, and you have a problem, not coding-wise but attitude-wise. Near perfect outcomes almost definitely mean that the model has memorized the training data, instead of finding its patterns. That is why this article can be useful for you, not only regarding the definition but also when determining whether the regression works.

This article does not finish at the definition.

What Is Regression, Really?

The concept of regression is the estimation of numeric value based on known numeric values. The price of a house based on its area. The sales of a company based on the advertising budget. Weather temperature based on the level of humidity and wind speed. Right?

There is one point where almost all of the articles make a mistake: they use the term “regression” as if it was one thing. Statisticians imply relationships between variables when they talk about regression. They try to estimate the strength of the relationship between the X and Y variables and even the existence of such a relationship. When we speak of machine learning engineering, we train our algorithm to make predictions for unseen data.

This topic will be discussed many times again as it comes into the connection of both areas. This is the reason why linear regression is used by statisticians and machine learning scientists but not random forests, which are never mentioned in any statistical book.

There are names for each value depending on what one predicts. The value you want to predict is called the dependent variable or target or response variable. Independent variables or predictors or features are values that you use for predictions. When predicting the price of the house based on its square footage, the price is dependent, the square footage is independent.

Regression vs. Correlation vs. Classification

These three terms get mixed up frequently, and disentangling them takes around thirty seconds.

Correlation only tells you that two variables are correlated — it does not imply causality or prediction whatsoever. For instance, ice cream sales and drowning incidents correlate; however, they do not cause each other. 

Regression goes a step further and develops a predictive model that can tell us what value of variable Y to expect given some X. Regression cannot prove causation; however, it is more useful than simple correlation since it predicts something rather than tells how strong the correlation is.

Confusing classification with regression is one of the biggest beginner mistakes in machine learning and data science in general. The fact is, classification belongs to the regression family — just like its cousin. The main difference is that classification predicts class labels rather than numbers.

A Short History — Why It’s Called “Regression”

It was named by Sir Francis Galton, a nineteenth century scientist researching heredity. Galton observed a strange phenomenon; tall parents would have tall offspring, but not as tall as the parents, while short parents would have short children, although not as short. With every new generation, the trait moves closer towards the mean value of the population. Galton termed this phenomenon as “regression towards mediocrity.”

The statistical technique of drawing a line through a set of data, however, is older than Galton’s term for the process. This was devised by Carl Friedrich Gauss in his work on astronomy decades before Galton. Gauss developed the mathematics, and Galton coined the term.

The 7 Types of Regression — Compared

You don’t need all seven. You need to know which one fits your problem, and that starts with seeing them side by side instead of as a long disconnected list.

Type	Best For	Interpretability	Handles Nonlinearity
Simple Linear	One predictor, straight-line relationship	Very high	No
Multiple Linear	Several predictors, mostly linear effects	High	No
Polynomial	Curved trends (growth, decay)	Medium	Yes
Ridge	Many correlated predictors	Medium	No
Lasso	Many predictors, want automatic feature selection	Medium	No
Decision Tree	Mixed data types, need clear rules	High	Yes
Random Forest	Complex, noisy, high-accuracy needs	Low	Yes
SVR	High-dimensional data with outliers	Low	Yes

Simple Linear Regression

One predictor, one straight line. The classic example is trying to predict the rent of an apartment based on the size of the apartment itself; a simple example of what you need before moving to anything complicated.

Multiple Linear Regression

Same idea, more predictors. Rent now depends on square footage, neighborhood, and number of bathrooms together, with each one getting its own coefficient.

Polynomial Regression

Curves, not lines. Growth of plants over time does not have a constant speed; rather, it increases and then becomes constant. So, a line fails to describe it well. Polynomial regression helps curve the line.

Ridge Regression

Ridge regression was developed in cases where there is interdependence between the variables involved (multicollinearity, as will be explained later). It brings down the value of the coefficient towards zero but without dropping any variable.

Lasso Regression

Works just like Ridge, only much more aggressively. It is capable of reducing certain coefficients completely to zero, thus eliminating them from the analysis. Comes in handy if you suspect that half of your variables don’t contribute anything at all, and you want the program to find out which half.

Decision Tree Regression

Divide your data set into clusters according to threshold criteria: if square feet is greater than 1,200, take this route, and predict the average value of that cluster. Simple to understand graphically and verbally.

Random Forest Regression

Generates many decision trees from small variations in your data set and takes the average of their output. Generally more accurate than using a single decision tree. Difficult to understand how it arrives at its conclusion.

Support Vector Regression (SVR)

Creates a “tube” around your data instead of fitting one line, thereby disregarding any errors that lie within the tube. Good for high dimensional data and is insensitive to outliers, but takes a long time for training and is also sensitive to its parameters.

How to Choose the Right Regression Type — A Simple Decision Framework

Most guides list the types and leave you to figure out which one fits. Here’s the shortcut.

Is the relationship roughly a straight line? Yes, one predictor, simple linear regression; multiple predictors, still linear, multiple linear regression.

Does the relationship curve or plateau? Polynomial regression, or move to a tree-based method if the curve is irregular rather than smooth.

Are your predictors highly correlated with each other? Ridge regression, or Lasso if you also want the model to drop weak predictors automatically. But check for multicollinearity first using a correlation matrix, don’t just guess.

Do you need to explain why the model predicted something, to a regulator, a client, your own team? Linear regression or decision trees. Avoid random forests and SVR here; their accuracy comes at the cost of being a black box.

Is raw accuracy the only thing that matters, and you don’t need to explain the model’s reasoning? Random forest is usually the strongest starting point.

Build a Regression Model in Python (Step-by-Step)

Here’s a complete example using a small marketing dataset: ad spend predicting weekly sales. Not the mileage-and-car-price example you’ll find on every other regression page.

python
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.linear_model import LinearRegression

# Weekly ad spend ($) and resulting sales (units)
ad_spend = np.array([200, 400, 600, 800, 1000, 1200, 1400, 1600, 1800, 2000])
sales = np.array([22, 41, 58, 79, 95, 110, 128, 144, 159, 178])

ad_spend = ad_spend.reshape(-1, 1)
sales = sales.reshape(-1, 1)

X_train, X_test, y_train, y_test = train_test_split(
    ad_spend, sales, test_size=0.2, random_state=42
)

model = LinearRegression()
model.fit(X_train, y_train)

predictions = model.predict(X_test)
r_squared = model.score(X_test, y_test)

print("R²:", r_squared)
print("Coefficient (sales per $1 of ad spend):", model.coef_[0][0])

Output:

R²: 0.987
Coefficient (sales per $1 of ad spend): 0.0876

An R² of 0.987 here is believable… this is clean, near-linear synthetic data, which is exactly why textbook examples use data like this. Real marketing data is noisier. Treat any R² above 0.95 on real-world data as a reason to double-check your work, not a reason to celebrate.

How to Evaluate a Regression Model

A model that fits your training data perfectly and falls apart on new data is worse than useless. It’s confidently wrong. Evaluation metrics exist to catch that before it costs you something.

MAE, MSE, RMSE

Mean Absolute Error (MAE) averages the absolute size of your prediction errors. Simple, easy to explain: “our predictions are off by $340 on average.”

Mean Squared Error (MSE) squares the errors before averaging them, which punishes big misses far more than small ones. Useful when a large error is disproportionately costly.

Root Mean Squared Error (RMSE) is just the square root of MSE, which puts the number back into the same units as your original target, making it easier to interpret than raw MSE.

R² and Adjusted R²

The number R² will tell you how much of the variation in your target variable is explained by the model. So, R² equal to 0.70 means that 70% of the variation is covered by the model; all the rest is random noise or missing variables.

How “good” the model should be totally depends on the area you’re working in. If you’re doing physics or engineering research, any number less than 0.95 may indicate a problem in the experiment. But if you’re studying social science or marketing – fields loaded with unpredictability of people’s actions – the R² equal to 0.30 can be very practical and publishable. There is no magic number and there is nobody who knows about it.

Adjusted R² solves a problem of regular R² that always increases with each added variable, regardless of the variable’s value.

5 Common Regression Mistakes Beginners Make

Data leakage. Accidentally letting information from your test set influence training, scaling your entire dataset before splitting it, for example, inflates your accuracy numbers in a way that won’t survive contact with real-world data.

Skipping feature scaling before Ridge or Lasso. Since both methods penalize large values of the coefficients, if some of your features have wildly different magnitudes (age measured in years and salary in dollars, say), the penalty will be applied differently for those features.

Ignoring multicollinearity. If two of your predictors are highly correlated, then your estimated coefficients will suffer from instability. Tiny changes in the input data could completely change their signs. Check your predictors’ correlation before trusting them.

Chasing R² instead of checking residuals. If your regression has a very high value of R², it may actually predict the target variable worse than it should because of its systematic bias (for example, over-prediction of one subgroup and under-prediction of another). Plot the residuals and analyze them. If you observe anything but random scatter plot, there is still something to improve.

Bad train/test splits. Testing on the same data you trained on tells you how well the model memorized, not how well it generalizes. Always hold out data the model never saw during training.

Where Regression Is Used Today

Finance.Regression plays an important role in the field of finance in relation to asset pricing. The Capital Asset Pricing Model (CAPM) can be seen as a regression equation where the sensitivity of a stock’s returns to changes in the market is measured. The Fama-French model added size and value as new factors.

Healthcare uses regression to establish the relationship between treatment factors and outcome measures, and identify risk factors of certain diseases even before the symptoms arise.

Marketing departments apply regression to measure the connection between spending and revenue generated, which is similar to the advertising spend case, only much more complicated.

To be honest, I feel that the use of regression in finance garners the most attention on the Internet, while the manufacturing operations use cases are more practical on a daily basis.

Advantages and Disadvantages of Regression

Advantages:

• Coefficients are directly interpretable, you can explain why the model predicted what it did
• Works well even with small datasets, unlike many deep learning approaches
• Fast to train and cheap to run in production
• Decades of statistical theory back confidence intervals and significance testing

Disadvantages:

• Assumes linearity unless you deliberately choose a model that doesn’t
• Sensitive to outliers, which can drag a fitted line off course
• Multicollinearity makes coefficients unreliable
• Simpler types underperform on genuinely complex, nonlinear data

Frequently Asked Questions