Exploratory Data Analysis (EDA)

Understanding Exploratory Data Analysis (EDA)

Exploratory data analysis (EDA) is a structured investigation of what a dataset contains, how it was collected, and which conclusions it can support. It combines data-quality checks, summaries, and comparisons to generate questions and identify limitations. EDA usually begins before modelling, but it also continues when new data or unexpected results reveal assumptions worth revisiting.

Start by defining the unit of observation: what does one row represent? An order, a customer, and a customer-month require different interpretations. Record units, collection dates, inclusion rules, and column meanings. A numeric customer identifier is not a measurement to average. Repeated customer identifiers might represent valid purchases rather than duplicate records. These distinctions determine whether later summaries answer your question.

Study individual variables before relationships. For numeric measurements, examine observed counts, missingness, center, spread, and extremes. The mean uses every value and is sensitive to unusually large observations; the median describes the middle ordered value. Quartiles summarize the distribution’s middle portion. For categories, inspect counts and proportions, including missing entries. A missing value means unknown or unavailable, not automatically zero.

Visualizations complement summaries because different distributions can share similar averages. Histograms reveal concentration and possible skew, although bin choices affect appearance. Scatterplots show relationships between numeric variables, while grouped summaries compare categories. Box plots flag relatively extreme observations, not necessarily errors. Examine suspicious records against source definitions before changing them. A large transaction could be a genuine bulk order rather than a misplaced decimal.

Relationships require context. A difference between groups may reflect their composition, collection periods, or other variables rather than an effect of group membership. Report denominators and missingness alongside averages. Correlation summarizes association, not causation; a weak linear correlation also does not rule out a curved relationship. Exploring many relationships makes chance patterns easier to find, so treat discoveries as hypotheses rather than confirmed findings.

When EDA informs predictive modelling, reserve the test set before outcome-driven exploration. Develop feature choices and cleaning rules using training data, and fit imputers or scalers only there. Keep the test set for final evaluation. For descriptive work, clearly state which population and period your observations actually cover.

Practical walkthrough

Consider a question: how do recorded order amounts differ between store and web purchases? Assume each row represents one distinct order and all amounts use the same currency. The small dataset below is deliberately constructed for inspection, not population inference. It requires Python with pandas installed and reads no external files.

First, establish structure. In a larger dataset, inspect df.shape, df.head(), and df.dtypes. Use df.info() directly: it prints a structural summary and returns None, so print(df.info()) adds an unnecessary None. Confirm that amounts are numeric and channel labels are consistent. Check uniqueness using the intended order identifier, not merely whether two amounts match.

Next, quantify missingness before choosing any treatment. The example counts missing amounts and reports their percentage of all orders. Leave the missing amount unknown: replacing it with zero would change the question from recorded spending to an invented assumption about spending. Compare missingness across channels because unequal coverage can affect comparisons.

Then examine center and range. Compare the overall mean with the median and inspect the maximum. Rather than deleting the largest order, flag it for source verification. A histogram would normally help inspect shape, but this exercise uses compact numerical summaries so its output is exactly reproducible.

Finally, group by channel. Report total rows, observed amounts, means, and medians together. The distinction between size and count exposes each average’s denominator. Finish with an interpretation and a follow-up question: web orders have a higher observed mean here, but is the largest amount valid, and why is another web amount missing?

Worked example

import pandas as pd

orders = pd.DataFrame({
    "order_id": [1, 2, 3, 4, 5, 6, 7, 8],
    "channel": ["store", "store", "store", "store",
                "web", "web", "web", "web"],
    "amount": [10, 12, 14, 16, 10, 12, None, 100],
})

amount = orders["amount"]
missing = int(amount.isna().sum())
print(f"Orders: {len(orders)}")
print(f"Missing amounts: {missing} ({100 * missing / len(orders):.1f}%)")
print(f"Observed mean: {amount.mean():.2f}")
print(f"Observed median: {amount.median():.2f}")
print(f"Observed range: {amount.min():.2f} to {amount.max():.2f}")

summary = orders.groupby("channel", sort=True)["amount"].agg(
    rows="size", observed="count", mean="mean", median="median"
)
print("channel | rows | observed | mean | median")
for channel, row in summary.iterrows():
    print(f"{channel} | {int(row['rows'])} | {int(row['observed'])} | "
          f"{row['mean']:.2f} | {row['median']:.2f}")

Expected output

Orders: 8
Missing amounts: 1 (12.5%)
Observed mean: 24.86
Observed median: 12.00
Observed range: 10.00 to 100.00
channel | rows | observed | mean | median
store | 4 | 4 | 13.00 | 13.00
web | 4 | 3 | 40.67 | 12.00

Seven amounts are observed, totaling 174, so their mean is 174 divided by 7, or 24.86 after rounding. The median is 12.00 because it is the fourth sorted observed value. The 100-unit order raises the overall mean substantially. Both channels contain four orders, but the web mean uses only three amounts. Web’s mean exceeds store’s while its median is lower, illustrating how the chosen summary changes interpretation. Neither summary establishes that purchasing online increases spending. The missing web amount and tiny sample prevent a confident generalization. Verify the unusual order and investigate missingness before recommending any operational change.

Common mistakes and how to avoid them

• Treating df.describe() as a complete audit.

  Add explicit missing-value counts, type checks, category frequencies, and identifier checks. Numerical summaries cannot establish whether units, row definitions, or collection coverage are correct.

• Deleting unusual values simply because they change the average.

  Check source records and plausible domain limits first. Preserve valid extremes; report mean and median together, and document any justified exclusion.

• Comparing group averages without checking coverage.

  Show total rows and nonmissing counts for each group. Investigate whether missing records differ systematically; observed-value averages may not represent all group members.

• Using test-set patterns to select features or fill missing values.

  Make these decisions on training data. Fit preprocessing within each training fold during cross-validation, then apply the fitted transformations unchanged to held-out data.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide