Feature Scaling

Understanding Feature Scaling

Feature scaling changes the numerical coordinates used to describe observations. Imagine comparing machines using operating time in hours and power consumption in watts. A difference of hundreds of watts can dominate a difference of several hours in Euclidean distance, even when both measurements matter. Scaling makes that balance an explicit modeling choice rather than an accidental consequence of measurement units. It does not discover which feature is important.

Scalers usually operate column by column: each feature gets its own transformation, learned across training observations. StandardScaler uses z = (x − mean) / standard deviation. With its default settings, each nonconstant training column has mean approximately zero and population standard deviation approximately one. Scikit-learn computes this deviation using division by n, not n − 1. A transformed value of 2 means two training standard deviations above that feature's training mean. Standardization does not make a skewed distribution Gaussian or remove outliers.

MinMaxScaler instead uses z = (x − minimum) / (maximum − minimum) for its default range. Nonconstant training columns map into [0, 1], with the observed endpoints becoming zero and one. Future observations can fall outside that interval because the training endpoints remain fixed. Both methods preserve ordering within a nonconstant column. Neither handles outliers robustly: extreme observations affect standardization statistics and can compress most min-max values into a narrow interval. RobustScaler, based on medians and quantiles, is an alternative worth validating.

Scaling matters for distance-based methods such as KNN, for many SVM configurations, and for optimization in neural networks. PCA centers features but does not automatically give them equal variance; scaling before PCA changes which directions count as high variance. Regularized linear models are also affected because coefficient penalties interact with feature units. Ordinary threshold-based decision trees generally do not need scaling: changing units preserves the candidate partitions determined by feature ordering.

Treat the scaler as learned model state. Its statistics must come only from training data, even though it never sees target labels. Otherwise, held-out information influences model construction. Scaling is not universally beneficial: equalizing variation can amplify noisy features or remove meaningful magnitude differences. Choose preprocessing using training-only validation, not repeated inspection of final test results.

Practical walkthrough

Begin with a two-dimensional numeric array: rows represent observations, and columns represent features. The example uses four deliberately simple training observations and one held-out observation. Assume complete, finite measurements with the same units and column order throughout. Prerequisites are basic Python arrays, means, and the training/test distinction. A suitable environment is Python 3.10+ with compatible NumPy 1.24+ and scikit-learn 1.3+ installed; the example requires no external data.

First, establish the split before learning any preprocessing statistics. Here, separate arrays make that boundary visible. In real work, select a split appropriate to the problem; time-ordered or grouped observations may require more care than random splitting. Keep the final test partition untouched during preprocessing selection.

Next, create StandardScaler and call fit_transform only on X_train. This combines learning column statistics with applying them to the training rows. Inspect mean_ and scale_ to connect the calculation to the original units. Then call transform on X_test using that same fitted object. The held-out row must use the training coordinate system, not a newly estimated one.

Repeat with MinMaxScaler to compare representations, not predictive accuracy. The held-out measurements intentionally exceed both training maxima. Their outputs demonstrate why the default range is a training-data property rather than a universal bound.

For model development, put the scaler before the estimator in a Pipeline. Cross-validation must fit that entire pipeline separately inside each training fold. At prediction time, pass raw features to the fitted pipeline. This keeps transformations consistent and avoids accidentally fitting preprocessing on validation folds before evaluating the model.

Worked example

import numpy as np
from sklearn.preprocessing import StandardScaler, MinMaxScaler

# Columns: operating time (hours), power consumption (watts).
X_train = np.array([
    [1.0, 100.0],
    [1.0, 300.0],
    [3.0, 100.0],
    [3.0, 300.0],
])
X_test = np.array([[5.0, 500.0]])

standard = StandardScaler()
train_standard = standard.fit_transform(X_train)
test_standard = standard.transform(X_test)

minmax = MinMaxScaler()
minmax.fit(X_train)
test_minmax = minmax.transform(X_test)

def show(label, values):
    print(f"{label}: " + ", ".join(f"{v:.2f}" for v in values))

show("Training means", standard.mean_)
show("Training scales", standard.scale_)
show("Standardized training means", train_standard.mean(axis=0))
show("Standardized training stds", train_standard.std(axis=0))
show("Standardized held-out row", test_standard[0])
show("Min-max held-out row", test_minmax[0])

Expected output

Training means: 2.00, 200.00
Training scales: 1.00, 100.00
Standardized training means: 0.00, 0.00
Standardized training stds: 1.00, 1.00
Standardized held-out row: 3.00, 3.00
Min-max held-out row: 2.00, 2.00

The training means are 2 hours and 200 watts. Each time value differs from its mean by one hour, and each power value differs by 100 watts, giving population standard deviations of 1 and 100. Standardization therefore maps the training entries to minus one or plus one. For the held-out row, (5 − 2) / 1 and (500 − 200) / 100 both equal 3. Min-max scaling instead gives (5 − 1) / (3 − 1) = 2, with the same result for power. Formatting fixes two decimal places. No model is trained, so these numbers demonstrate transformations, not performance improvements.

Common mistakes and how to avoid them

• Fitting separate scalers on training and test data.

  Fit once on training data and reuse transform. Separate fits assign different meanings to identical transformed values and use information that should remain held out.

• Expecting every transformed dataset to have mean zero or stay within [0, 1].

  These properties describe fitted, nonconstant training columns. Inspect future values without refitting. MinMaxScaler's clip=True enforces bounds but discards information about how far values exceed them.

• Scaling category codes as though they were measurements.

  Encode nominal categories appropriately, often with one-hot encoding. Use ColumnTransformer to scale selected numeric columns; arbitrary category identifiers have no meaningful numerical distance.

• Centering a large sparse feature matrix.

  Use StandardScaler(with_mean=False) when preserving sparsity. Subtracting means would turn many implicit zeros into nonzero entries; without centering, do not expect zero column means.

Review questions

Further reading

• scikit-learn user guide