Data Analysis with Python

Eighty percent of a data analyst’s time is spent cleaning data, not analyzing data. Tutorials never cover this topic because they don’t show you how to clean your data. Instead, they show you how to create a scatter plot without showing you how to convert your salary field from text strings to numeric values – so, the scatter plot you make will be wrong.

This tutorial won’t do that to you.

You’ll learn everything you need to analyze a dataset: asking the question, importing the data, cleaning the data, exploring the data, visualizing the data, modeling the data, and answering your question. NumPy, pandas, Matplotlib, Seaborn, and scikit-learn will all be used. However, the emphasis will be on decision-making rather than programming. Do you impute the missing value, or delete the row? Is pandas no longer the right tool for the job? 

Do you switch from analysis to communication?

These are the questions that distinguish good analysts from mediocre ones.

What Data Analysis with Python Actually Means

It is safe to say that “data analysis” is used in reference to all kinds of activities, from counting rows in a spreadsheet to training neural networks. But before you start writing your code, the difference needs to be clear.

Data analysis is the practice of analyzing existing data in order to answer a particular business question. You are not making forecasts for the future; you are not setting up pipelines for future predictions. You are answering questions such as “Which product line has the highest return rate?” or “Is there a correlation between tenure and satisfaction score?” There is no need to collect additional data because you already have all the data you need.

Data analysis differs both from data science and business intelligence. The former includes machine learning techniques used to create predictive models based on input data.

The awareness of that boundary prevents one from over-analyzing something which is supposed to be analyzed within three hours.

Pandas were created by Wes McKinney in 2008 when he worked for AQR Capital Management. The reason was that he needed a better way of financial time series manipulation in Python. This background is important because pandas initially solved a practical problem.

Why Python and Not R or Excel?

The R programming language is ideal for statistical analysis. Excel is efficient for handling small and static datasets. Python has three practical reasons for being better.

One, the libraries provide an end-to-end solution for your analysis process without the need to switch platforms. Two, Python allows connectivity to databases and APIs, allowing data to be pulled from any source. Three, Python skills transfer to automation and data engineering in case your career evolves that way.

According to the latest Stack Overflow Developer Survey, Python topped the list in data-related applications for the third year in a row. Job listings for analysts requiring Python are also more than those requiring R across all markets.

This is where the jobs are. This is why Python is better.

What You’ll Actually Be Able to Do After This

Four tangible results, not just skills.

You will go from a dirty CSV file with missing data, incorrect datatypes, and duplicate entries to a clean data set. You will perform an exploratory analysis that reveals insights the business can use. You will create a graph that even a non-technical manager can understand in ten seconds. And you will know when your analysis is finished.

This last one takes more practice than writing code.

The Python Data Analysis Stack: What Each Tool Does

The key point here is that you will not be proficient with all libraries before starting. You learn about the capabilities of each library to know what to use when.

NumPy provides operations on arrays. Numerical calculations on large sets of numbers. You will be using it mostly indirectly, since pandas works on top of NumPy.

pandas is the place where most of the analytical computations take place. It manipulates structured and labeled data – the kind of data found in spreadsheets and relational databases. DataFrames are named tables and they provide a rich set of functions to filter, group, reshape, and join your data.

Matplotlib is the base library used for plotting. Verbose, but each and every component of the plot is controllable. When the accurate placement is critical, Matplotlib is the way to go.

Seaborn builds upon Matplotlib. It allows to create statistical plots with less code, with default parameters tuned for statistics, and with grouping based on the hue parameter.

scikit-learn is used for modeling. Specifically for analysis tasks, you will need regression methods to quantify dependencies and predict results.

And finally, one more library to consider in 2026: Polars. It is a pandas alternative developed in Rust that processes large datasets significantly faster than pandas . For data under a million rows, pandas is fine. Beyond that scale, Polars is increasingly part of how analysts work.

Setting Up Your Environment

This step is missing from practically every single tutorial out there. This is why most people get stuck even before they start their analysis.

Install Python 3.11 or 3.12. If you want to do data work, I recommend using conda instead of pip because it solves issues related to dependencies. If you use pip, create a virtual environment first:

bash

python -m venv analysis-env
source analysis-env/bin/activate    # Mac or Linux
analysis-env\Scripts\activate       # Windows

Install the core stack:

bash
pip install numpy pandas matplotlib seaborn scikit-learn jupyterlab

To the editor: Jupyter Notebook is great for exploratory purposes. However, when you need version control and file organization, VS Code with the Jupyter extension is ideal. Choose whichever you prefer, but try not to switch from one environment to another within the same project.

The 5-Step Workflow Most Tutorials Don’t Show You

This is a pretty self-evident point, until you try doing it without a proper process. Coding blindly generates results answering a different question.

Below you have an outline following CRISP-DM, or the Cross-Industry Standard Process for Data Mining, which is the process adopted by most professional data analysis teams. It is not a linear process. You will cycle through these steps. This is not because you made a mistake; this is how data analysis works.

Step 1: Define the Question – before even opening Python. Step 2: Get the Data – import it into Python. Step 3: Clean the Data – fix things that are broken. Step 4: Analyze and Visualize – discover what the data tells us. Step 5: Communicate the Results – make the answer actionable.

Steps 3 and 4 cycle. You clean, find an anomaly while analyzing, return to cleaning, analyze again. Expect to spend at least two or three iterations on real data sets.

Step 1: Write the Question in One Sentence Before You Open Python

This is how analysts waste time.

The marketing team wants you to “take a look at the campaign data.” You spend three hours creating pivot tables and graphs. You show them. They say: “That’s great, but what we really wanted to know was whether email beats paid social media for people under 35.”

Three hours wasted. Not because your analysis is bad. Because nobody wrote down what they actually wanted to know.

An effective analytical question should be clear, answerable using available information, and linked to an important decision that needs to be made.

Not good: “Analyze our sales data.”

Better: “Is there a statistically significant difference in the average order value between customers obtained via organic search compared to those who clicked on paid ads in Q1 2026?”

The latter clearly indicates which metrics to consider, which test to perform, and what the answer to the question will look like. If you cannot formulate your question in one sentence, your analysis isn’t even ready to begin.

Acquiring Your Data: Loading Files into Python

Oddly enough, this is the part that is covered most hurriedly in two lines of code in any tutorial. Their data-cleaning portion doesn’t make sense since they never showed us what the original data looks like.

python
import pandas as pd

df = pd.read_csv("sales_data.csv").convert_dtypes()
print(df.head())
print(df.info())

Let’s begin with CSV – the file format you will encounter most frequently:

.convert_dtypes() tells pandas to pick the best data type for each column. Without it, integer columns containing even one missing value default to float64, which creates subtle downstream problems. Use it every time you load.For Excel files:

python
df = pd.read_excel("sales_report.xlsx", sheet_name="Q1_2026")

For a SQLite database:

python
import sqlite3

conn = sqlite3.connect("company.db")
df = pd.read_sql_query("SELECT * FROM orders WHERE year = 2026", conn)
conn.close()

Once you have done the cleaning of the data, it is recommended that you save it in the form of an Apache Parquet file instead of a CSV file because it saves all the data exactly in the way it is typed, makes efficient use of compression techniques, and is loaded much faster into pandas. Apache Parquet was created by Twitter and Cloudera in 2013.

python
df.to_parquet("cleaned_data.parquet")
df = pd.read_parquet("cleaned_data.parquet")

CSV resets your data types on every load. Parquet does not. That single difference saves hours across a long project.

Free Datasets to Practice On Right Now

Don’t waste days looking for the “perfect” dataset. Here are four to use.

Kaggle at kaggle.com/datasets: the biggest repository of real-life datasets, with many having community notebooks that showcase alternative analyses of the same data.

Our World in Data at ourworldindata.org: well-curated datasets on economic and demographic trends globally. Great for time series analyses.

data.gov: open datasets from the US government. Realistically messy and great for practicing cleaning skills.

UCI Machine Learning Repository at archive.ics.uci.edu: academic benchmark datasets. The famous adult income and housing datasets are often used for EDA practice.

Choose one of these datasets. Do the entire process of data exploration on it before moving to the next one. Frequent switching between datasets is the fastest way to learn syntax while neglecting analysis.

Cleaning Your Data with pandas: The Step That Decides Everything

It must be admitted that nobody likes data cleaning. However, the quality of your cleaning will influence the quality of every single answer you give afterwards. It’s better to remain silent than to give a wrong answer with such confidence that someone will do something based on your analysis.

In most papers related to data engineering, it is stated that analysts spend up to 80 percent of their time preparing data for the analysis. This percentage looks unrealistic until you analyze data from a client’s database for the first time.

Remember to always work with a copy of the file. Don’t even think about manipulating the original data file.

python
df = pd.read_csv("raw_data.csv").convert_dtypes()
clean_df = df.copy()

If your analysis breaks halfway through, df needs to be exactly what you loaded. The .copy() is not optional.

Handling Missing Values: Three Decisions, Not One

There are three options for dealing with missing data, and each is suitable under certain conditions.

Drop the row if the missing value is in a crucial column and cannot be imputed effectively. It’s okay to discard 2 percent of rows, but discarding 30 percent of rows means you’ve altered the fundamental nature of your data set.

Impute using statistics if the column is numeric. Impute using median if the distribution is skewed: income, sales, housing prices, all columns where outliers affect the average more than the typical value. Impute using mean if the distribution is symmetric.

Impute using mode if the column is categorical and one value significantly outweighs the others.

python
# See what's missing and where
print(clean_df.isnull().sum())

# Fill salary column with median
clean_df["salary"] = clean_df["salary"].fillna(clean_df["salary"].median())

# Drop rows missing a critical identifier
clean_df = clean_df.dropna(subset=["customer_id"])

This decision between the three choices is more important than the actual coding. Plugging in the mean for an unbalanced salary range will increase your estimate of the “average” employee. The median is always a better choice for salaries.

Fixing Data Types: The Problem You See Constantly

Currency columns stored as strings are the most common data type error in real-world datasets. They look like this: “$45,200.00”. Python reads that as text. Math cannot be done on text.

python
clean_df["revenue"] = (
    clean_df["revenue"]
    .str.replace("[$,]", "", regex=True)
    .astype("Float64")
)

Dates stored as strings are the second most frequent problem. A column containing “March, 2025” will not sort correctly and will not support date arithmetic until it is converted:

python
clean_df["sale_date"] = pd.to_datetime(clean_df["sale_date"])
clean_df["sale_year"] = clean_df["sale_date"].dt.year

One error is responsible for a disproportionate share of beginner debugging time: the SettingWithCopyWarning. It appears when you try to modify a slice of a DataFrame rather than the DataFrame itself. Pandas is warning you that your change may not be saved. The fix is to use .loc[] for targeted updates or to make sure you’re working on an explicit .copy() throughout.

Detecting Outliers: Investigate Before Deleting

Outliers are not necessarily wrong. The salary of a company’s CEO in a data set on salaries is a valid outlier. A movie with a duration of 1,200 minutes is a mistake. Examine everything before deleting.

IQR approach – skewed distribution:

python
Q1 = clean_df["salary"].quantile(0.25)
Q3 = clean_df["salary"].quantile(0.75)
IQR = Q3 - Q1

outliers = clean_df[
    (clean_df["salary"] < Q1 - 1.5 * IQR) |
    (clean_df["salary"] > Q3 + 1.5 * IQR)
]

Z-score method — roughly normal distributions:

python
import numpy as np

mean = clean_df["salary"].mean()
std = clean_df["salary"].std()
clean_df["z_score"] = (clean_df["salary"] - mean) / std

outliers = clean_df[clean_df["z_score"].abs() > 3]

After identifying them: go back to the source. If the value is genuinely extreme but accurate, keep it and note it. If it’s a data entry error, fix or drop it. Never delete on assumption alone.

EDA: Finding What the Data Is Actually Saying

However, in practice, EDA is usually considered merely a formality in most tutorials, in which case you do a quick .describe() and then move on to the regression analysis. It should be the other way around because this is the phase in which thinking actually takes place.

I have seen analysts spend two whole days meticulously preparing their cleaning script and then just five minutes on exploration before proceeding to modeling.

The exploration phase, however, is when the thinking occurs. Bypassing this stage leads to developing very accurate models for the wrong questions — the best way to develop an analysis that is completely useless for anyone.

This stage is all about hypothesis generation, trying to find out what patterns exist, what variables correlate, and how unexpected relationships need further investigation.

This is where you start every time you work with a new dataset:

python
print(clean_df.info())       # Column types and non-null counts
print(clean_df.describe())   # Statistical summary
print(clean_df.head(10))     # Actual values

The .describe() function provides the count, mean, standard deviation, minimum value, and the 25th, 50th, and 75th percentiles for each numerical column. One of the best indicators from this function is the difference between the median and the mean. When there is a big difference, then the distribution is skewed.

Correlation Analysis: The Most Misused Tool in Data Analysis

Articles about this topic get the perspective completely backwards. Most articles explain how to do correlations and show the strong ones. This is not an end in itself. It is a question.

The correlation between ice cream sales and drowning is actually 0.85. It is measurable, real, and entirely meaningless because both of these things are caused by the weather. The correlation does not give us any idea about the underlying process. Every correlation raises a question: what third factor might be causing both of these?

python
correlation_matrix = clean_df.corr(numeric_only=True)

Visualize it:

python
import seaborn as sns
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(10, 8))
sns.heatmap(
    correlation_matrix,
    annot=True,
    fmt=".2f",
    cmap="coolwarm",
    ax=ax
)
ax.set_title("Feature Correlation Heatmap")
plt.tight_layout()
plt.show()

A practical rule of thumb: correlations greater than 0.7 or less than -0.7 can be explored further. Correlations ranging from -0.3 to 0.3 are weak and do not usually form a good basis for a hypothesis.

Correlation is a question, not an answer. This one shift in mindset will completely change your results section.

Grouping and Aggregation: Comparing Segments of Data

.groupby() answers one question better than almost any other tool: does this metric differ meaningfully across groups?

Say you have a dataset of job postings from early 2026 and you want to know median salary by job category:

python
salary_by_role = (
    clean_df
    .groupby("job_category")["salary"]
    .agg(["mean", "median", "count"])
    .sort_values("median", ascending=False)
    .reset_index()
)

print(salary_by_role)

Calculate both the average and the median. When there is a wide difference between the two statistics for a specific category, then that category has a skewed salary distribution, which means something structurally important about compensation in that industry.

Do not merely provide the figures. When there is a 40 percent difference between the medians of two job categories, it is more than a figure, but an indicator of something structural in the labor market. What is that?

Visualizing Data with Python: Charts That Make a Point

That’s kind of the whole point of a chart: not to show data, but to make one specific, arguable claim about what the data means. A chart without a “so what” sentence is decoration.

Pick the chart type that matches the relationship you want to show:

Chart type	When to use it
Scatter plot	Relationship between two numerical variables
Bar chart	Comparing values across categories
Histogram	Distribution of a single numerical variable
Heatmap	Correlations or patterns across many variables

Four types. They cover 90 percent of what analysts actually need to communicate.

Matplotlib: Precise Control at the Cost of Verbosity

python
import matplotlib.pyplot as plt

fig, ax = plt.subplots(figsize=(9, 5))

ax.scatter(
    clean_df["years_experience"],
    clean_df["salary"],
    alpha=0.6,
    edgecolors="white",
    linewidths=0.4
)

ax.set_title("Salary vs. Years of Experience", fontsize=14, fontweight="bold")
ax.set_xlabel("Years of Experience")
ax.set_ylabel("Annual Salary (USD)")

plt.tight_layout()
plt.show()

Label every axis. Add a title to every chart. A chart without labels is useless to anyone who wasn’t in the room when you built it — and that includes you three weeks from now.

Set alpha on scatter plots. When points overlap, fully opaque markers hide where the density actually is. alpha=0.6 shows clustering without obscuring it.

Seaborn: More Information, Less Code

Seaborn’s real value is the hue parameter. It encodes a third variable through color without you writing a loop or a manual legend.

python
import seaborn as sns

fig, ax = plt.subplots(figsize=(9, 5))

sns.scatterplot(
    data=clean_df,
    x="years_experience",
    y="salary",
    hue="job_category",
    palette="tab10",
    alpha=0.7,
    ax=ax
)

ax.set_title("Salary vs. Experience by Job Category")
plt.tight_layout()
plt.show()

Three variables in one chart. That’s a significantly richer picture than two variables, and it took fewer lines of code. Use Seaborn for grouped and distributional visualizations. Use Matplotlib when precise layout control is needed. In practice, both appear in the same notebook.

Building a Regression Model with scikit-learn

Regression gets filed under “data science” by most tutorials. But analysts are asked to run it constantly. “What salary should we expect for someone with eight years of experience in this role?” is a regression question, not a machine learning question.

Linear regression finds the mathematical relationship between an input and an output, then uses that relationship to estimate values for inputs it hasn’t seen.

python
from sklearn.linear_model import LinearRegression

X = clean_df[["years_experience"]]   # Two-dimensional input required
y = clean_df["salary"]

model = LinearRegression()
model.fit(X, y)

r_squared = model.score(X, y)
coefficient = model.coef_[0]
intercept = model.intercept_

print(f"R-squared:   {r_squared:.2f}")
print(f"Coefficient: {coefficient:.2f}")
print(f"Intercept:   {intercept:.2f}")

R-squared measures what fraction of salary variation is explained by years of experience. An R-squared of 0.72 means 72 percent of salary variation is captured by that single variable; the remaining 28 percent comes from industry, location, company size, and individual negotiation.

Higher is not always better. An R-squared of 0.99 almost always means you accidentally included a variable that is just another form of your target — a phenomenon called data leakage. Models that score perfectly on existing data typically fail on new data.

Plot actual values against predicted ones to see where the model fits and where it misses:

python
predictions = model.predict(X)
sorted_X = X.sort_values("years_experience")

fig, ax = plt.subplots(figsize=(9, 5))
ax.scatter(X, y, alpha=0.5, label="Actual", color="steelblue")
ax.plot(
    sorted_X,
    model.predict(sorted_X),
    color="tomato",
    linewidth=2,
    label="Predicted"
)
ax.set_xlabel("Years of Experience")
ax.set_ylabel("Salary (USD)")
ax.legend()
plt.tight_layout()
plt.show()

Where actual values cluster far from the predicted line, the model is missing something. That gap is a hypothesis for the next analysis.

The Five Mistakes That Cost Beginners the Most Time

Correct. This is one section that is not usually covered in any data analysis tutorial. But it should be included since these five mistakes make up a huge percentage of the time that beginners spend debugging in search of an error that isn’t there.

Mistake 1: Altering the Original DataFrame

You perform a few cleaning operations on the data frame df. The results of your analysis are off. You check your data, only to realize that the original df has been modified.

python
clean_df = df.copy()
# Everything from this point uses clean_df only

Never

Mistake 2: Visualizing before cleaning

You import your data, throw up a scatterplot, see a trend, and lock yourself into a hypothesis. Until you realize the trend was caused by a salary listed as $1,200,000 rather than $120,000. Outliers and data type issues create false patterns. Clean first, then visualize.

Mistake 3: Correlation = Causation

Already addressed in the EDA section, but worth a second mention because this error pops up time and time again when analysts present findings to executives. Just because two variables have a strong correlation doesn’t mean they’re related. Your next sentence after identifying correlation should always be “which could be caused by…”

Mistake 4: Cleaning without EDA – or EDA before knowing what data you have

Spending two full days developing an intricate data cleaning process. Then, one hour into EDA, discovering that your data set lacks the required column. You’re not even able to analyze the data. Perform a crude first clean. Get to EDA as soon as possible. Adjust your clean based on actual discovery, not future discovery.

Mistake 5: Scrolling past the SettingWithCopyWarning

Python logs it as a warning, not an error. Beginners scroll past it. But this warning often means your DataFrame changes are not being saved, and you’ll spend an hour debugging a problem that exists in how you’re writing to the DataFrame, not in the data itself.

Communicating Findings to Non-Technical Stakeholders

Three rules. Absolutely no padding.

State your findings first, not your methods. People don’t care that you used .groupby() with multi-level aggregation. They need to know that the East region did not hit its Q1 targets by 23 percent, and that the decrease began during week two. Findings come first, and methods come second if people ask.

Only one chart per claim. Five charts in sequence, and people won’t remember any of them. One chart. One claim. One sentence explaining the meaning of the chart and what needs to be done about it.

Put the “so what?” sentence below each chart. Not the explanation of the chart, but the implication. For example: “Email creates 2.4 times more valuable orders on average than paid social among customers under 35 years old, which means that re-allocating the budget toward email would increase ROI while staying within budget.” Without that sentence, a chart is just an image.

What to Learn Next: A Realistic Roadmap

Here is a timeline based on how things really work, rather than what the landing page of a course promises.

Stage 1: 4 to 8 weeks of steady practice. You can import, clean, and perform EDA on your own without errors. You generate four chart types. You comprehend what correlation tells you and what it doesn’t. You’ve completed the tasks mentioned above using at least two different data sets selected by yourself, not provided through a tutorial.

Stage 2: 2 to 4 months after completing Stage 1. You’ve completed the entire workflow from start to finish using unfamiliar data. You’ve created and interpreted a linear regression. You presented results to a non-technical person and answered any follow-up questions without breaking the chain. You’ve worked in at least three industries or domains.

Stage 3: 3 to 6 months after completing Stage 2. You know when pandas slow down and have tried Polars as an alternative. You’ve written reusable functions rather than copying code cell-by-cell across multiple notebooks. You know about DuckDB and know when to use it instead of importing all the data into pandas just to query it.

Where these skills lead in the US market as of 2026:

Role	Typical US Salary Range
Data Analyst	$65,000 to $95,000
Business Intelligence Analyst	$70,000 to $105,000
Data Scientist	$95,000 to $140,000+

None of these require a computer science degree. All of them require deliberate, consistent practice on real data. Not more tutorials.

Frequently Asked Questions