Handling Missing Data

Understanding Handling Missing Data

A missing measurement is not a measurement of zero. For example, an unrecorded apartment area means “unknown,” whereas zero square metres has a different meaning. In numeric NumPy arrays, np.nan commonly represents missingness. Many scikit-learn estimators reject these values, although some, including histogram-based gradient boosting, support them. Missing-data handling therefore depends on both the data’s meaning and the chosen estimator, rather than a universal requirement to remove every NaN.

Start by asking why values are absent. A sensor failure, an optional questionnaire field, and a measurement collected only for selected cases create different problems. Missingness may relate to other features or to the unobserved value itself. Neither deleting records nor filling blanks automatically removes the resulting bias. Document what is known about collection, and avoid interpreting a convenient replacement as a recovered observation.

Dropping rows is simple but discards their observed information too. It can also change which cases remain represented. Dropping a largely empty column may be reasonable when its useful information is limited, but missingness percentage alone is not enough to decide. Imputation instead retains rows by replacing missing entries according to a rule learned from observed training values.

SimpleImputer treats columns independently. Mean imputation uses a numeric column’s average; median imputation uses its middle observed value and is less sensitive to extreme observations. Most-frequent imputation can handle categorical values, while constant imputation can assign an explicit “unknown” category. These approaches are inexpensive baselines, not reconstructions of reality. Repeated replacements can reduce variation and distort relationships between features. More elaborate methods introduce additional assumptions and computational cost.

A missingness indicator preserves whether a value was originally absent. With add_indicator=True, SimpleImputer appends indicators for columns containing missing values during fitting. Such flags may help prediction, but improvement is not guaranteed, and changing collection practices can undermine them.

Finally, imputation learns from data even without labels. Compute replacement values using training data only, then reuse them unchanged for validation, testing, and prediction. During cross-validation, fit preprocessing separately inside each training fold. The scikit-learn guide’s discussions of imputation, pipelines, and leakage support treating this transformation as part of the fitted learning procedure.

Practical walkthrough

Use Python 3.10 or later with NumPy and scikit-learn installed; this example uses APIs available in scikit-learn 1.2 and later. It creates everything in memory. No dataset download or trained prediction model is needed because the goal is to inspect the transformation itself.

Imagine two apartment features: area in square metres and building age in years. The arrays below are already separated into training and held-out test records. This fixed separation makes the arithmetic reproducible; it is not a recommendation to split real datasets by position. Real splits must respect relevant time ordering or shared entities, such as repeated observations of the same building.

First, count NaNs by column in the training array. Each feature has one missing entry. Confirming the representation matters: text such as “unknown” would not automatically behave like np.nan in this numeric example.

Next, create a median imputer with missingness indicators. Calling fit_transform on training data calculates and stores two medians, fills training gaps, and appends two indicator columns. Read statistics_ to inspect the learned replacements rather than guessing them from the completed table.

Then call transform, not fit_transform, on the test array. The test apartment with area 200 must not influence the learned area median. Inspect the output order: area, age, area-missing flag, age-missing flag. An indicator of 1 means the original entry was absent; 0 means it was observed.

For subsequent model selection, place the imputer before the estimator in a Pipeline and cross-validate that complete pipeline. Compare strategies using training-side validation, keeping the final test set reserved for evaluation.

Worked example

import numpy as np
from sklearn.impute import SimpleImputer

# Columns: area in square metres, building age in years.
X_train = np.array([
    [40.0, 10.0],
    [60.0, np.nan],
    [np.nan, 30.0],
    [80.0, 20.0],
])
X_test = np.array([
    [np.nan, 50.0],
    [200.0, np.nan],
])

imputer = SimpleImputer(strategy="median", add_indicator=True)
train_ready = imputer.fit_transform(X_train)
test_ready = imputer.transform(X_test)

print("Training missing counts:", np.isnan(X_train).sum(axis=0).tolist())
print("Learned medians:", imputer.statistics_.tolist())
print("Columns: area, age, area_missing, age_missing")
print("Training rows:")
for row in train_ready.tolist():
    print(row)
print("Test rows:")
for row in test_ready.tolist():
    print(row)

Expected output

Training missing counts: [1, 1]
Learned medians: [60.0, 20.0]
Columns: area, age, area_missing, age_missing
Training rows:
[40.0, 10.0, 0.0, 0.0]
[60.0, 20.0, 0.0, 1.0]
[60.0, 30.0, 1.0, 0.0]
[80.0, 20.0, 0.0, 0.0]
Test rows:
[60.0, 50.0, 1.0, 0.0]
[200.0, 20.0, 0.0, 1.0]

Observed training areas are 40, 60, and 80, so their median is 60. Observed ages are 10, 30, and 20; sorting gives a median of 20. Only missing entries receive these replacements. Both features were incomplete during fitting, so both receive appended indicators, represented here as floating-point zeros and ones. The observed test area of 200 stays unchanged, and the missing test age becomes 20. Notice that an observed area of 60 and an imputed area of 60 have different flags. This deterministic output verifies preprocessing behavior; it provides no evidence that median imputation improves a model’s predictive performance.

Common mistakes and how to avoid them

• Imputing the whole dataset before splitting, or refitting on the test set.

  Split first and fit only on training records. Reuse transform for held-out data. Put preprocessing inside the cross-validated pipeline so each fold learns its own replacements.

• Replacing every blank with zero because the estimator requires numbers.

  Check what zero means in each feature. Use a justified training-derived replacement or an explicit missing category; retain an indicator when the distinction deserves evaluation.

• Assuming every input column survives median imputation.

  Check for entirely missing training columns. No median exists for them, and default handling can remove them. Choose an explicit policy; consider keep_empty_features=True when preserving dimensionality is required.

• Dropping incomplete feature rows without updating their labels.

  Apply the same row mask to features and targets. Inspect which cases are excluded, and define how prediction will handle incomplete incoming records rather than silently ignoring them.

Review questions

Further reading

• scikit-learn user guide