Exploratory Data Analysis Using Python

⏱ Estimated reading time: 3 min

Exploratory Data Analysis (EDA) is the process of examining and understanding a dataset before applying machine learning or statistical models. It helps identify patterns, detect anomalies, test hypotheses, and check assumptions. Python, with libraries like Pandas, NumPy, Matplotlib, and Seaborn, is widely used for performing EDA efficiently.

1. Importance of EDA

EDA helps in:

• Understanding data structure

• Identifying missing values

• Detecting outliers

• Finding relationships between variables

• Summarizing statistical properties

• Selecting the right model for ML

• Transforming data for better insights

2. Steps in EDA Using Python

Step 1: Importing Libraries

import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns

Step 2: Loading the Dataset

df = pd.read_csv("data.csv")
df.head()

head() displays the first five rows, useful for initial understanding.

Step 3: Understanding the Structure of Data

Check shape:

df.shape

Summary of columns:

df.info()

Statistical summary:

df.describe()

This gives mean, median, std, quartiles, min, and max.

Step 4: Handling Missing Values

df.isnull().sum()

Ways to fix missing data:

• Fill with mean/median:

  df['age'].fillna(df['age'].mean(), inplace=True)
  

• Drop missing values:

  df.dropna(inplace=True)
  

Step 5: Detecting Outliers

Using boxplot

sns.boxplot(df['salary'])

Using IQR method

Q1 = df['salary'].quantile(0.25)
Q3 = df['salary'].quantile(0.75)
IQR = Q3 - Q1

outliers = df[(df['salary'] < Q1>1.5*IQR) | (df['salary'] > Q3 + 1.5*IQR)]

Step 6: Univariate Analysis

For numerical data:

sns.histplot(df['age'], kde=True)

For categorical data:

df['gender'].value_counts().plot(kind='bar')

Step 7: Bivariate Analysis

Numerical vs Numerical → Scatter Plot

sns.scatterplot(x='age', y='income', data=df)

Categorical vs Numerical → Box Plot

sns.boxplot(x='gender', y='salary', data=df)

Correlation Heatmap

sns.heatmap(df.corr(), annot=True, cmap='coolwarm')

This highlights relationships between variables.

Step 8: Multivariate Analysis

Pairplot

sns.pairplot(df)

This visualizes interactions between multiple features.

Step 9: Feature Engineering During EDA

• Creating new features

• Converting categorical data using encoding

• Scaling numerical values

• Removing irrelevant features

Example:

df['income_per_age'] = df['income'] / df['age']

Step 10: Final Summary of Insights

After completing EDA, you prepare a summary containing:

• Key statistics

• Data quality issues

• Outliers detected

• Important correlations

• Trends and patterns

• Suggestions for preprocessing

This becomes the foundation for model building.

Conclusion

EDA using Python is a crucial step in the data analysis process. It helps transform raw data into meaningful insights. Python libraries like Pandas enable efficient data cleaning and manipulation, while visualization libraries like Matplotlib and Seaborn help uncover hidden patterns. Proper EDA leads to better modeling decisions and improved machine learning performance.