Exploratory Data Analysis

EDA is the process of examining a dataset before modelling to understand its structure, quality and patterns. It involves summary statistics, checking data types, finding missing values and outliers, studying distributions and relationships between variables, and visualising the data. EDA guides data cleaning, feature engineering and the choice of model, and often reveals business insights on its own.

1. Understand the business problem and the data dictionary 2. Load the data and check shape, columns and data types 3. Check missing values and duplicates 4. Univariate analysis: distributions of each variable 5. Bivariate/multivariate analysis: relationships between variables and with the target 6. Detect outliers and anomalies 7. Check correlations 8. Summarise insights and decide on cleaning and feature engineering

• df.shape — number of rows and columns • df.head() / df.tail() / df.sample() — look at rows • df.info() — column types and non-null counts • df.describe() — summary statistics (include='all' for categoricals) • df.isnull().sum() — missing values per column • df.duplicated().sum() — duplicate rows • df['col'].value_counts() — category frequencies • df.nunique() — number of unique values per column

First understand why data is missing (random, or meaningful like 'no second phone number'). Options: • Drop rows or columns with too many missing values • Impute numeric columns with mean (no outliers) or median (skewed data) • Impute categorical columns with mode or an 'Unknown' category • Forward/backward fill for time series • Model-based imputation (KNN imputer, iterative imputer) • Add an 'is_missing' indicator column when missingness itself is informative

Outliers are values that are unusually far from the rest of the data. Detection methods: • Box plots and visual inspection • IQR rule: values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR • Z-score: |z| > 3 • Scatter plots for multivariate outliers • Algorithms like Isolation Forest or DBSCAN for complex data

• Investigate first — they may be data-entry errors or genuine rare events (e.g., fraud) • Correct or remove clear errors • Cap/winsorize extreme values at a percentile (e.g., 1st and 99th) • Apply transformations such as log or square root to reduce skew • Use robust methods (median, tree-based models) that are less sensitive • Keep them if they are meaningful for the business question

• Univariate: analyses one variable at a time — histograms, box plots, value counts, mean/median. • Bivariate: studies the relationship between two variables — scatter plots, correlation, grouped bar charts, cross-tabs. • Multivariate: examines three or more variables together — pair plots, heatmaps, colour/size-encoded scatter plots, or dimensionality reduction like PCA.

• Distribution of a numeric variable: histogram, KDE plot, box plot • Category counts: bar chart (or count plot) • Numeric vs numeric: scatter plot • Numeric vs category: box plot or violin plot • Trend over time: line chart • Correlation between many numeric variables: heatmap • Parts of a whole (few categories): stacked bar; use pie charts sparingly

Skewness measures the asymmetry of a distribution. Right (positive) skew has a long tail of high values (e.g., income); left (negative) skew has a long tail of low values. In skewed data, the mean is pulled toward the tail, so the median is often a better summary. Handling: log, square-root or Box-Cox/Yeo-Johnson transformations, or using models that are robust to skew.

Kurtosis describes the 'tailedness' of a distribution compared with a normal distribution. High kurtosis (leptokurtic) means heavy tails and more extreme values/outliers; low kurtosis (platykurtic) means light tails. A normal distribution has kurtosis 3 (excess kurtosis 0). It helps assess the risk of extreme values, for example in finance.

Correlation means two variables move together; causation means one directly causes the other. Correlation can arise from coincidence or a hidden third variable (confounder) — e.g., ice-cream sales and drowning incidents both rise in summer because of hot weather. Establishing causation requires controlled experiments (A/B tests) or careful causal-inference methods.

• Pearson correlation measures the strength of a linear relationship between two continuous variables and is sensitive to outliers. • Spearman correlation measures a monotonic relationship using ranks; it works for non-linear but consistently increasing/decreasing relationships, ordinal data and data with outliers. Both range from −1 to +1.

Detect with df.duplicated().sum() for exact duplicates, or df.duplicated(subset=['email']) for duplicates on key columns. Inspect them before deleting to make sure they are real duplicates, then remove with df.drop_duplicates(), choosing which record to keep (keep='first' or based on the latest timestamp).

Wrong data types cause wrong analysis — for example, numbers stored as strings can't be summed, dates stored as text can't be sorted by time, and IDs stored as numbers may be averaged by mistake. Check with df.dtypes or df.info() and convert using pd.to_numeric(), pd.to_datetime() or astype('category').

• Frequency counts and percentages with value_counts(normalize=True) • Bar charts/count plots • Check cardinality (number of unique categories) and group rare categories into 'Other' • Cross-tabulation (pd.crosstab) with the target or other categories • Compare numeric variables across categories using groupby or box plots • Chi-square test to check association between two categorical variables

• Convert the date column to datetime and set it as the index • Plot the series over time to see trend and seasonality • Resample to daily/weekly/monthly aggregates • Use rolling averages to smooth noise • Decompose into trend, seasonality and residual (seasonal_decompose) • Check for missing dates, outliers and structural breaks • Look at autocorrelation (ACF/PACF) plots

• Mean: arithmetic average; best for symmetric data without outliers. • Median: middle value when sorted; robust to outliers and skew (e.g., salaries, house prices). • Mode: most frequent value; useful for categorical data. In a right-skewed distribution, typically mean > median > mode.

IQR (Interquartile Range) = Q3 − Q1, the range of the middle 50% of the data. Values below Q1 − 1.5×IQR or above Q3 + 1.5×IQR are flagged as outliers (this is what box-plot whiskers show). In pandas: Q1 = df['x'].quantile(0.25); Q3 = df['x'].quantile(0.75) IQR = Q3 - Q1 outliers = df[(df['x'] < Q1 - 1.5*IQR) | (df['x'] > Q3 + 1.5*IQR)]

Multicollinearity occurs when two or more features are highly correlated with each other. It makes linear model coefficients unstable and hard to interpret (it matters less for prediction with tree models). Detection: correlation heatmap (e.g., |r| > 0.8) and the Variance Inflation Factor (VIF > 5–10 indicates a problem). Fixes: drop one of the correlated features, combine them, or use regularization or PCA.

• pandas: loading, cleaning and summarising data • NumPy: numerical operations • Matplotlib and Seaborn: static visualisations • Plotly: interactive charts • SciPy/statsmodels: statistical tests • Automated EDA tools: ydata-profiling (formerly pandas-profiling), Sweetviz, D-Tale

A correlation heatmap displays the correlation coefficients between all pairs of numeric variables using colours. It quickly shows which features are strongly related to the target (useful for feature selection) and which features are highly correlated with each other (possible multicollinearity). Remember it only captures linear (Pearson) or monotonic (Spearman) relationships, not causation.

• Visual: histogram/KDE plot (bell shape) and Q-Q plot (points close to the diagonal line) • Summary: skewness close to 0 and excess kurtosis close to 0 • Statistical tests: Shapiro-Wilk (smaller samples), Kolmogorov-Smirnov, Anderson-Darling With large samples, tests flag tiny deviations, so visual checks matter too.

groupby splits data into groups and applies aggregations, e.g., df.groupby('city')['sales'].agg(['mean','sum','count']), revealing how a metric differs across segments. Pivot tables (pd.pivot_table) summarise data across two dimensions, e.g., average sales by region (rows) and month (columns). Both are essential for finding segment-level patterns and business insights.

Check the target distribution with df['target'].value_counts(normalize=True) and a bar chart. If one class is far more common (e.g., 98% non-fraud vs 2% fraud), the dataset is imbalanced. This matters because models can achieve high accuracy by predicting only the majority class, so you must choose appropriate metrics (precision, recall, F1, PR-AUC) and consider resampling or class weights.

Data profiling is systematically summarising a dataset's structure, content and quality: row/column counts, data types, missing and unique values, distributions, min/max, common values and potential data quality issues. Tools like ydata-profiling generate a full HTML report in one line of code, which is a fast starting point for EDA.

It examines how each feature relates to the target variable to find the most predictive features. For a numeric target: scatter plots and correlations. For a categorical target: box plots of numeric features by class, grouped means, and conversion rates by category (e.g., churn rate by contract type). It guides feature selection and engineering.

1. Check the churn rate (target balance). 2. Inspect data types, missing values and duplicates. 3. Univariate: distributions of tenure, monthly charges and categorical plan types. 4. Bivariate: churn rate by contract type, payment method, tenure bands and support calls; compare charges of churned vs retained customers with box plots. 5. Check correlations and multicollinearity. 6. Identify outliers. 7. Summarise insights such as 'month-to-month customers in their first 6 months churn the most' and recommend actions.

Binning (discretisation) groups continuous values into intervals, e.g., age into 18–25, 26–35, 36–50. It helps reveal patterns (e.g., churn by tenure band), reduces the effect of noise and outliers, and makes results easier to explain to business users. In pandas use pd.cut (fixed ranges) or pd.qcut (equal-sized quantile bins).

EDA is about understanding the data — exploring distributions, relationships, patterns and problems. Data cleaning is fixing the problems found — handling missing values, removing duplicates, correcting data types, treating outliers and standardising formats. In practice they are iterative: EDA reveals issues, cleaning fixes them, and further EDA confirms the data is ready.

• Start with the business question and the 3–5 key insights • Use simple, clearly labelled charts (bar and line charts over complex plots) • Quantify findings in business terms (e.g., '35% of churn comes from first-year customers') • Highlight data quality caveats briefly • End with recommended actions and next steps • Use dashboards (Power BI, Tableau) or a short slide deck rather than raw notebooks