Exploratory Data Analysis

Francis Anscombe, in 1973, created four datasets in order to make a point. They have identical meanings. Identical variances. Identical coefficients. Identical linear regressions. Identical R² of 0.67. Input each of them into any statistical software, and you will get the same results. Identical numbers. Same model. Same conclusions.

Draw them, and they all become different. The first dataset shows a simple relationship between X and Y — the regression model is valid. In the second dataset, there is no relationship at all, just a curve. Linear regression cannot possibly apply. The third dataset is a nearly perfect linear relationship with a tiny point throwing off all calculations. The fourth dataset features a perfectly identical series of points in X values, with the only different point making a regression line completely incorrect.

Four plots. One point. The whole rationale behind Exploratory Data Analysis explained without using math.

Here comes the thing that most EDA guidebooks don’t mention: the consequences. They explain what EDA is without mentioning the risk of skipping on the technique. That’s why, before moving on to how-to articles featuring Python codes, we need to understand the purpose of EDA clearly.

What Is Exploratory Data Analysis?

Exploratory data analysis is defined as the act of looking into the data set either visually or through statistical analysis without having developed any models yet. This helps us to understand the nature of the data and see if there are any issues, as well as discovering some patterns and making certain conjectures.

The idea was coined by John Tukey, an American mathematician who worked primarily at Bell Labs and Princeton. He published a book entitled “Exploratory Data Analysis” in 1977 and stated that the field had become too narrow-minded in focusing mainly on confirmatory analysis.

This notion remains relevant. In fact, it has become increasingly important over time.

The datasets obtained from Internet of Things (IoT) sensors, large language model embeddings, behavioral monitoring, and satellite imagery in 2026 come as one dataset from various sources, with no neat structure, obvious connections between variables, and documented data-generating process. Exploratory data analysis allows you to gain insight into what kind of data you are dealing with before choosing an inappropriate modeling approach.

Exploratory data analysis differs from descriptive statistics and inferential analysis.

EDA vs. Descriptive Statistics vs. Inferential Analysis

	EDA	Descriptive Statistics	Inferential Analysis
Goal	Understand data, discover surprises	Summarize known patterns	Test hypotheses on populations
Tools	Pandas, Seaborn, Plotly, visualization	Mean, median, std dev	t-tests, confidence intervals
Output	Visualizations, hypotheses, questions	Reports, dashboards	p-values, predictions
When to use	Before any modeling — always first	Once patterns are identified	After hypotheses are formed

EDA sits upstream of both. It shapes what the other two can responsibly answer. You cannot choose the right statistical test without knowing your distribution. You cannot form a meaningful hypothesis without knowing what the data actually contains.

Why EDA Is the Most Underrated Step in Data Science

Data scientists spend roughly 79% of their time on data work: 60% in data cleansing and structuring and 19% in data acquisition, leaving about 20% of the time available for analysis and modeling activities. This statistic comes from CrowdFlower’s Data Science Report in 2016 and has been generalized in the subsequent years to “80%”. According to the Anaconda State of Data Science Report, the combined figure has been estimated to be closer to 50%.

The statistic appears in all data science articles. However, few people wonder why this happens.

This is not because of inefficiency but due to the complexity of EDA. The impact of bad EDA only manifests in production.

Let us revisit Anscombe’s Quartet. Fit a linear regression model for all four data sets. You will receive the same result: slope = 0.5, intercept = 3, R² = 0.67. Everything is perfectly identical. Based on the statistics alone, you will use the linear model for all four data sets. This would be a mistake in the case of three out of four. Not an error in the sense that the results might differ slightly from the truth, but an error serious enough that you need to rethink the whole approach to modeling.

What Happens When You Skip EDA

Two problems emerge invariably. Both of them.

Firstly, your modeling is based on a flawed foundation. Imbalanced distribution skews model coefficients. Outliers pull regression lines in the direction that misleads them. Multicollinearity among features leads to unpredictable behavior when one feature moves beyond the domain seen during training. None of these issues are present when you look at accuracy on a training set. You discover all of them once the model goes to production.

The second problem has its roots in the lack of exploration. Not knowing what you do not know may be disastrous. For example, your retail company could have developed a churn prediction model without realizing that 12% of “loyal customers” had not placed any orders for the last year and half, even though they never formally churned according to your CRM data. You train a predictive model based on an incorrect signal. Your tests show impressive results, but your model is inaccurate in production right where you need it most.

In some cases, your best result from EDA is to walk away from this project. Either collect more data or explain why this dataset will not work for you.

The 8-Step EDA Process (With Python Code)

Point to keep in mind before moving ahead: EDA is an iterative process. You’ll go back.

A finding made in step 5 will bring you back to missing value treatment in step 3. This doesn’t mean there’s anything wrong with the process; on the contrary, it means that the process is functioning well. Anticipate this right from the start; it will save you a lot of heartburn.

Step 1 — Define Your Questions Before Touching the Data

Avoid opening the dataset at all costs.

This is the step that data scientists tend to skip, assuming they are wasting their time. This is the step that will save you from wasting months. Write down 3–5 concrete questions to ask of the data, and do not open a single row to answer them. Not “understand the data” as a question; more like, “Which customer cohorts have monthly churn greater than 15%?” Or, “Do readings from sensors measuring vibrations correlate with the failure of the bearing in 24 hours?”

Now write down explicit assumptions. “We assume that the data from Q3 2024 and afterwards captures behavior of our clients after migration to a new platform.” “We assume the ‘inactive’ flag means no logins for 30 days, not no purchases.” Get them on record; they will be tested, potentially falsified during EDA.

The biggest pitfall here is allowing yourself to formulate questions from the data itself. This is how you let confirmation bias into your analysis right from the beginning.

Step 2 — Load and Inspect Your Dataset

Six lines tell you more than most people realize:

import pandas as pd

df = pd.read_csv('your_dataset.csv')

print(df.shape)               # rows × columns — first sanity check

print(df.info())              # column names, dtypes, non-null counts

print(df.head())              # first 5 rows — see what you're working with

print(df.describe())          # summary stats for numerical columns

print(df.isnull().sum())      # missing values per column

print(df.duplicated().sum())  # exact duplicate rows

What you’re looking for: Object dtypes where they should have been float64, which indicates there’s been some form of non-numeric input in your numeric columns. Unexpectedly high null counts in columns you thought were fully populated. Columns where the min/max values from the describe() method make no sense for your domain.

Age with a min value of −3 and a max value of 847 doesn’t indicate an issue with the dtype. It indicates a data quality issue. And you just found it in less than 30 seconds.

Step 3 — Handle Missing Data

Missingness is not random.

There are three processes by which missing data may occur, and the nature of each determines the strategy that should be used for dealing with missingness:

• MCAR: Missing Completely at Random: There is no connection between the missingness and any variable in the data set. A faulty sensor was responsible for missing data points; the missingness is not systematically connected to the remaining observations. Imputation is safe.
• MAR: Missing At Random: The missingness is correlated with other observable variables, but not the missing value itself. There are more missing income data points among younger survey respondents, but given the age variable, the missingness is still random. Imputation is possible using the correlated variables.
• MNAR: Missing Not At Random: The missingness is correlated with the missing value itself. The wealthiest survey respondents do not answer the income questions. The sicker patients do not provide lab measurements. It is dangerous; imputation will introduce systematic biases.

import seaborn as sns

import matplotlib.pyplot as plt

# Visualize the missing data pattern

sns.heatmap(df.isnull(), cbar=False, yticklabels=False, cmap='viridis')

plt.title('Missing Data Heatmap')

plt.show()

# Impute numerical columns with median (appropriate for MCAR)

df['feature'].fillna(df['feature'].median(), inplace=True)

# KNN imputation for MAR — uses related variables to estimate missing values

from sklearn.impute import KNNImputer

imputer = KNNImputer(n_neighbors=5)

df_imputed = pd.DataFrame(

    imputer.fit_transform(df[numerical_cols]),

    columns=numerical_cols

)

A new approach that is good to be aware of: the employment of a language model to make inferences about likely values based on contextual field information around them. This is still in development and not yet considered best practice – the results are nondeterministic and difficult to validate. Good for specific applications; this does not replace all other methods of imputation.

Step 4 — Univariate Analysis

Look at each variable alone before looking at them together. You need a baseline understanding of every column’s behavior before you can meaningfully interpret how columns relate.

For numerical variables:

import matplotlib.pyplot as plt

import seaborn as sns

# Distribution shape — histogram

df['feature'].hist(bins=30, edgecolor='black')

plt.title('Distribution of Feature')

plt.show()

# Spread and outliers — box plot

sns.boxplot(y=df['feature'])

plt.show()

# Key statistics

print(df['feature'].describe())

print(f"Skewness:  {df['feature'].skew():.2f}")

print(f"Kurtosis:  {df['feature'].kurt():.2f}")

If the skewness coefficient is greater than 1 or less than -1, then the variable requires a logarithmic transformation prior to modeling. If the kurtosis coefficient is greater than 3, it indicates heavy tails, meaning there are more extreme values than what a normal distribution would produce.

Categorical Variables:

# Counts and proportions

df['category'].value_counts().plot(kind='bar')

plt.title('Category Distribution')

plt.show()

print(df['category'].value_counts(normalize=True))

If the skewness coefficient is greater than 1 or less than -1, then the variable requires a logarithmic transformation prior to modeling. If the kurtosis coefficient is greater than 3, it indicates heavy tails, meaning there are more extreme values than what a normal distribution would produce.

Categorical Variables:

# Counts and proportions

df['category'].value_counts().plot(kind='bar')

plt.title('Category Distribution')

plt.show()

print(df['category'].value_counts(normalize=True))

The key takeaway from all of this: an extreme skew towards one side of a target variable. When your classification target consists of 95% class A, your model appears 95% accurate but is completely ineffective since the model simply predicts class A for every data point. Spot this bias in your exploratory data analysis or down the road when you’ve deployed your model.

Step 5 — Bivariate and Multivariate Analysis

Now you look at how variables interact.

# Full correlation matrix

plt.figure(figsize=(12, 8))

sns.heatmap(df.corr(), annot=True, cmap='coolwarm', center=0)

plt.title('Correlation Matrix')

plt.show()

# Two numerical variables, colored by target

sns.scatterplot(x='tenure', y='monthly_charges', hue='churn', data=df)

plt.show()

# Multiple relationships at once — useful for smaller feature sets

sns.pairplot(df[['age', 'tenure', 'charges', 'churn']], hue='churn')

plt.show()

Make sure you have this written in a clear manner somewhere on your EDA sheet: correlation creates hypotheses. Correlation doesn’t prove causation. There could be another third variable that is driving both the variables in correlation. EDA creates the questions; inference solves them.

With datasets having vector embeddings — which will be common in 2026 when teams use foundation models to create embeddings of text, products, and behavior — add the following in addition to the regular correlation matrix:

from sklearn.metrics.pairwise import cosine_similarity

similarity_matrix = cosine_similarity(df[embedding_columns].values)

sns.heatmap(similarity_matrix, cmap='Blues')

plt.title('Embedding Similarity Matrix')

plt.show()

Step 6 — Outlier Detection and Treatment

This is where the judgment needs to be applied and not the statistician: Outliers may not necessarily be mistakes.

If we consider the case of fraud detection, here the anomaly itself is the key message — the whole process is designed to detect it. In the example of using statistics in predictive maintenance in manufacturing, the abnormal vibration before the breakdown of a bearing would be the most important piece of information in the whole dataset.

Statistical analysis first; then judgment.

# IQR method — univariate

Q1 = df['feature'].quantile(0.25)

Q3 = df['feature'].quantile(0.75)

IQR = Q3 - Q1

lower_bound = Q1 - 1.5 * IQR

upper_bound = Q3 + 1.5 * IQR

outliers = df[(df['feature'] < lower_bound) | (df['feature'] > upper_bound)]

print(f"Outliers detected: {len(outliers)} rows")

# Isolation Forest — multivariate outlier detection

from sklearn.ensemble import IsolationForest

iso = IsolationForest(contamination=0.05, random_state=42)

df['outlier_flag'] = iso.fit_predict(df[numerical_cols])

# Returns: 1 = inlier, -1 = outlier

Every detected outlier must first be checked based on domain expertise prior to taking any steps. An amount of $50,000 within a set of transactions where each transaction amounts to $200 may just be a mistake in data entry, or it could simply be the largest purchase by the company’s most valuable customer.

Step 7 — Data Transformation and Feature Engineering

EDA tells you what the data needs. This step acts on that information.

import numpy as np

from sklearn.preprocessing import MinMaxScaler

# Log transformation — corrects right-skewed distributions

# log1p handles zero values safely (log(1 + x))

df['log_feature'] = np.log1p(df['feature'])

# Min-max scaling — brings features to the same range [0, 1]

scaler = MinMaxScaler()

df[['scaled_feature']] = scaler.fit_transform(df[['feature']])

# One-hot encoding — converts categorical variables for modeling

df = pd.get_dummies(df, columns=['category_column'], drop_first=True)

# Derived features — built from EDA insights, not from guesswork

df['charge_per_tenure_month'] = df['monthly_charges'] / (df['tenure'] + 1)

df['is_high_value'] = (

    (df['monthly_charges'] > df['monthly_charges'].quantile(0.9)) &

    (df['tenure'] > 24)

).astype(int)

Derived features are where domain knowledge and EDA insights meet. You would not have come up with charge_per_tenure_month by analyzing the summary statistics. You came up with it because Steps 4 and 5 indicated that each feature was correlated with churn separately, but their ratio is likely a better predictor than each of the individual columns.

That is the distinction between creating features based on an analysis of your dataset versus creating features using a how-to guide.

Step 8 — Document and Communicate Your Findings

Almost all EDA guidelines cease their existence prior to this step.

An EDA report is not a Jupyter notebook with 40 plots. Instead, it is a comprehensive document that a non-technical stakeholder is able to read within 15 minutes and fully comprehend. The following six components need to be included in your EDA report:

1. Data overview — number of rows and columns, data source, date range covered by the dataset, data types, and potential limitations. Anyone reading this section should be able to get a full picture of which dataset you are using without opening it.
2. Important distribution insights — key takeaways from performing univariate analysis. Which distribution is characteristic for the target feature? Which of the features have highly skewed distributions? Which ones appear to be close to constant and cannot serve as valuable predictors?
3. Detected anomalies — outliers and other data quality issues, with row count and examples specified. Do not just write “several outliers were detected”. State clearly “47 rows have transaction_amount greater than $10,000 and are to be analyzed by a business expert”.
4. Identified relationships — top 3–5 relationships between features and supporting visualizations.
5. Data Quality Log – every decision made in the EDA recorded explicitly. Strategy to handle missing data, treatment of outliers, columns dropped, reasoning behind dropping. This forms the audit trail to ensure reproducibility.
6. Modeling hypotheses – 3–5 hypotheses generated from EDA. These form the initial hypotheses for choosing models and features. Write complete sentences: “Customers that downgrade their subscription plan in months 2–4 are more likely to churn in 90 days.”

The notebook is where you record your process. The report is what you need to submit. Not the same document, making them one is the reason for every stakeholder confusion.

When Is EDA Complete?

Strangely enough, there is rarely an actual answer to this question.

“Until you understand the data” is not a stopping criterion. Here are five which are:

1. Visualization and analysis of every variable’s distribution
2. Accounting for any missing data, along with the reasoning for each choice
3. Review of the correlation matrix, plus examination of every |r| > 0.7 pair of variables
4. Generation of at least 3–5 hypotheses for testing, written out in clear English
5. Completion of the data quality log, with reasons for every decision taken

Satisfy all five of those, and go back to see how your answers to the business questions in Step 1 stack up. Are your questions now answerable, or has it been demonstrated that they aren’t? Then EDA is over.

For production pipelines, however, it never ends. That will get its own section later on.

EDA Tools and Libraries for 2026

The right tool is the one your team will actually use — not the most sophisticated option available.

Python EDA Libraries

Python remains the go-to choice for conducting data science within most teams by 2026.

Here is what each library does and when to use it:

Pandas is the starting point for any EDA workflow. Loading, filtering, grouping, cleansing, and all summarization steps – this is where you start with data science in Python. It is an essential package that cannot be bypassed during EDA in Python.

Matplotlib provides the core visuals functionality. The syntax is rather verbose, yet precise. When there is no specific function in Seaborn or you need total control over every element in your figure, Matplotlib will come to your rescue.

Seaborn provides statistical visualizations using Matplotlib. With more straightforward syntax and default settings optimized for EDA visualizations, it includes such types of figures as heat maps, distributions, scatter plots with color coding, and violin plots. Usually, Seaborn figures are faster to generate and look neater than Matplotlib charts.

Plotly creates interactive plots which allow users to explore the data through hovering, zooming in, and filtering. If you present results to stakeholders who must independently explore the insights without a data scientist, switch from Seaborn to Plotly.

ydata-profiling (a.k.a. pandas-profiling) creates an HTML EDA report in just five lines of code:

import pandas as pd

from ydata_profiling import ProfileReport

df = pd.read_csv('your_dataset.csv')

profile = ProfileReport(df, title="EDA Report", explorative=True)

profile.to_file("eda_report.html")

That report covers distributions, correlations, missing data, duplicates, and variable interactions across every column. A junior analyst would need a full day to build it manually. Use it as a starting point — not as a substitute for the EDA process itself. The report shows you what to look at; it doesn’t tell you what any of it means.

Library	Primary Use	Best For	Difficulty
Pandas	Data manipulation	Every project	Beginner
Matplotlib	Custom static charts	Precise control	Intermediate
Seaborn	Statistical visualization	Standard EDA plots	Beginner
Plotly	Interactive charts	Stakeholder-facing demos	Beginner–Intermediate
ydata-profiling	Automated EDA report	First pass on any new dataset	Beginner
Sweetviz	Comparison EDA	Train vs. test set analysis	Beginner
AutoViz	Automatic chart selection	Quick exploration sprints	Beginner

R for EDA

R remains the best tool for statisticians and teams focused on research work. Python would be ideal for data science teams close to the engineering space. That’s not about preference, but rather stems from the very purpose of each language. For now, there’s no need to move to Python for EDA purposes if you’re working in R.

library(tidyverse)     # Hadley Wickham's ecosystem: ggplot2, dplyr, tidyr

library(DataExplorer)  # Automated EDA for R

# Single-function automated EDA report

DataExplorer::create_report(df)

ggplot2, written by Hadley Wickham, is one of the best data visualization tools out there, especially in terms of elegance. This comes from its use of the Grammar of Graphics concept, where you layer your data, aesthetic attributes, and geometry. You end up with professional-grade plots with fewer lines of code compared to most other Python tools. ggplot2 is better for conventional statistical visualization than Matplotlib.

Automated and No-Code EDA Tools

In reality, however, these are merely faster; they are not meant to do the work for you.

Tableau Prep and Power BI Dataflows cater to organizations that already exist within these ecosystems. Graphical, drag-and-drop interfaces are great for exploring distributions and mapping relationships. However, it is not recommended for custom statistical analysis or feature engineering with Python.

Hex and Deepnote are both collaboration-first notebook tools. Real-time editing, stakeholder-level sharing without knowing Python, and automated version control. By 2026, they have rendered email-based notebook sharing obsolete in medium-sized data teams.

The principle is: automation makes reporting fast. Automation does not know whether an outlier was a simple mistake or your most valuable customer. Business context requires human insight, and no automation can substitute; eventually, it will give you confidence and be wrong.

EDA Visualization Best Practices

The right chart is the one that answers your specific question — not the one that looks most sophisticated.

Match the graph to its corresponding data type. Distributions: histogram for shape; box plot for spread and outliers; violin plot if you want both at once. Relationship: scatter plot between two continuous variables; heatmap for correlation matrices. Categorical: bar chart for frequencies; grouped bar chart when you’re making comparisons by a second variable. Time series: line chart; seasonal decomposition if you can identify periodicity.

Use color with purpose. Sequential schemes (light-to-dark) indicate magnitude. Diverging schemes (blue-white-red) indicate deviation from a central point. Qualitative schemes denote unordered categorical differences. Combining the conventions incorrectly, such as applying a sequential scheme to categorical data or a diverging scheme to basic frequency counts, is actually misleading the reader. It’s an incredibly common mistake when producing EDA.

Interactivity beats static visuals when you need your stakeholders to drill down. With Plotly visualizations, they can hover over individual data points to get exact values, zoom into areas of interest, and filter results by a categorical value without requiring a data scientist. With static graphics, you become the chokepoint for any further analysis of your EDA findings.

The insight should be annotated; just annotating the data alone won’t cut it. An unannotated graph demonstrating an unusual peak in correlation is relying on readers’ interpretation skills alone. “Churn rate is three times higher for customers in months 2 through 4 of their tenures” would be the right text annotation. This text explains why the graph was made.

Common EDA Mistakes (And What They Cost You)

Articles related to the topic mostly get it wrong here. They treat the methodology errors as ignorance of how to do it, when in fact most of the problems can be categorized as judgment errors — which aren’t resolved through additional knowledge of Python.

Confirmation bias – seeking out support for your beliefs

The belief was put into your head that customer satisfaction will result in more revenue, so naturally, that’s all you test. You plot satisfaction and revenue, find the correlation, and believe the results support your hypothesis, but you never tested to see if purchase frequency, service level, or type of product might better predict results.

Over-cleaning — cleaning out the signal along with the noise

The presence of outliers makes the data noisy. Noisy data makes modeling more difficult. And that is why data cleaning involves removing outliers automatically. But when it comes to fraud detection, medical anomalies, and industrial faults, doing so is entirely counterproductive, as the outlier itself is what you are looking for, and learning such an algorithm will only fail to recognize the actual event.

P-hacking — testing relationships until significance appears

If you perform enough correlations between pairs of variables, one will turn out to be statistically significant simply by coincidence. This is even more likely in the context of EDA, since there is the greatest inclination to explore every conceivable correlation. How to deal with this problem? Register your hypotheses in Step 1.

Not accounting for how the data was generated

The format in which survey data and transaction data appear in a dataframe is the same. But they are two very different data types. Survey data is based on self-reporting and is prone to social desirability bias, and may not be representative. Transaction data is based on true behaviors. Using the two interchangeably during the EDA phase leads to incorrect conclusions because they are measuring two entirely different variables.

Viewing EDA as a point-in-time activity

This one is expensive. Conducting EDA once upfront without following up is the assumption that data will remain static over time. Data will never remain static, and in many cases, it becomes increasingly dynamic over time. Different customer acquisition strategies lead to new behaviors. IoT sensor drift causes baseline levels to drift over time. Seasonality causes changes in distribution of values you did not account for in your initial analysis.

EDA Applications by Industry

Finance – fraud detection and credit risk. EDA identifies unusual transaction velocities, geographic inconsistencies, and behavioral abnormalities when compared to an established pattern of this particular account. It comes down to this – what is normal for this particular account? Everything is visible through the prism of this norm. A single transaction is not suspicious; a single transaction coming from another country at 3 a.m. of four times the value than the account average is a bunch of anomalies that EDA detects prior to modeling.

Healthcare – patient outcome prediction. EDA uncovers problems with data collection practices across different hospitals, missing laboratory results that exhibit characteristics of MNAR missingness (patients who are sicker are missing lab values more often – a pattern in itself) and issues with clinical trials where demographic imbalance leads to non-generalizable models.

Retail/E-commerce – churn and segmentation. EDA uncovers the difference between customers who did not purchase for a while but are still active versus those who actually churned. This is only noticeable after comparing their transaction records against other sources of information such as web logins, email click-through rates and support tickets. Without this insight, retention marketing efforts will be targeted at the wrong customers.

Manufacturing – Predictive maintenance. Time Series EDA concentrates on time-dependent patterns – does the vibration rise steadily before the bearing fails, or does it rise abruptly? Is there a correlation between abnormal temperatures and specific shifts, specific operators, or conditions of upstream machinery? It’s the difference between employing a time series model, threshold alerts, or multi-variate anomaly detection – three entirely distinct methods.

Marketing – Attribution modeling. The EDA shows that the team’s last-touch attribution model used for three years assigns all the credit to the channel that completes the conversion but disregards the other six touchpoints that led the customer to that decision. The information is apparent in EDA without developing an attribution model – whether anyone notices it is another matter.

EDA in Production ML Pipelines

Exploratory Data Analysis (EDA) performed within a notebook is a single analysis process, while EDA in a production environment is an ongoing analysis process.

After deployment, the model will receive inputs that differ from the data it was trained on. The customer behavior will change. The baseline of the sensors will shift over months. New product categories will be introduced to the catalog, violating the distribution assumptions the model had during training.

Great Expectations is the industrial-strength software suite used to validate datasets automatically. It allows users to define certain expectations about their data like column X meaning being in the range of 18-80, having no nulls in column Y, and column Z having data only from the specified set. Once defined, Great Expectations takes care of these validations automatically for each new dataset:

import great_expectations as ge

context = ge.get_context()

validator = context.get_validator(

    datasource_name="production_datasource",

    data_asset_name="customer_features"

)

validator.expect_column_mean_to_be_between("age", min_value=18, max_value=80)

validator.expect_column_values_to_not_be_null("customer_id")

validator.expect_column_values_to_be_in_set(

    "subscription_tier", value_set=["basic", "standard", "premium"]

)

validator.save_expectation_suite()

Drift detection catches when the statistical properties of incoming data shift significantly from the training distribution — a signal that the model may no longer be reliable:

from scipy import stats

# Kolmogorov-Smirnov test — compares two distributions

statistic, p_value = stats.ks_2samp(

    training_data['feature'].dropna(),

    new_batch_data['feature'].dropna()

)

if p_value < 0.05:

    print(f"Significant drift detected (p={p_value:.4f}) — review model performance")

In the context of the Cross-Industry Standard Process for Data Mining (CRISP-DM), which is a methodology for carrying out data mining projects, EDA is applied mainly during the phase of Data Understanding, which comes immediately after Business Understanding. The CRISP-DM model has the highest recognition among data scientists compared to other frameworks, although agile methods are incorporated into the model phases in production settings rather than being followed in sequence. In current times, Data Understanding and Data Preparation have a loop that runs continuously.

EDA Checklist — Before You Call It Done

Pre-Analysis

• Business questions documented (3–5, specific and testable)
• Assumptions written down explicitly before opening the dataset
• Data sources and provenance understood
• Expected data types per column noted prior to loading

In-Analysis

• Shape, dtypes, and missing values summarized (Step 2)
• Mechanism of missing data evaluated for each column (MCAR / MAR / MNAR)
• Approach to handling missing data described and executed
• Graphical representation of distributions generated for all important variables
• Skewness and kurtosis verified; transformations considered
• Correlation matrix analyzed; all r-values above 0.7 examined
• Outliers statistically determined and evaluated based on subject matter expertise
• Class imbalance confirmed for the target variable
• Feature engineering suggestions made based on EDA results

Post-Analysis

• EDA report complete (all six sections — see Step 8)
• 3–5 testable modeling hypotheses written in plain English
• Data quality log complete with all decisions recorded and reasoned
• Open questions listed for stakeholder review
• All transformations documented for reproducibility

Frequently Asked Questions

Conclusion

However, there are some aspects of EDA that are destined to evolve. As Python libraries continue to be updated, new automated EDA packages arise while others become obsolete. What might be considered best practices in 2026 will no longer apply two years down the road, and anyone claiming otherwise is peddling their wares.

The nature of the problem solved by EDA, however, remains constant.

Data will be messier than expected, relationships will be more complex than described by summaries, as evidenced by Anscombe’s work in 1973, and any model based upon such data is bound for failure.

I believe that the typical portrayal of EDA — as a preliminary activity, as something that must be done, but as the tedious and necessary background to the actual process of modeling — does actual harm to good data science. It incentivizes fast results over comprehension. It views exploration as overhead rather than as the basis upon which all other steps must be properly grounded.

And as we approach 2026, where LLMs create synthetic datasets, where AI helpers highlight anomalies before anyone can blink, where inputs come through multimodal data from sensors and vision models… judgment becomes even more critical, not less. The problem isn’t whether the tool saw a pattern; it’s whether the tool interpreted it correctly — and whether that interpretation still stands six months from now.

Start by working off the list above. Do it for your next project before writing any code. The datasets you want to fool you always appear clean.