PCA (Dimensionality Reduction)

Understanding PCA (Dimensionality Reduction)

Principal component analysis (PCA) replaces a set of numeric features with new coordinates called principal component scores. Imagine plotting two sensors whose readings usually rise together. The points form a narrow diagonal cloud. Rather than storing both coordinates, PCA can describe each point mainly by its position along that diagonal. This is feature extraction, not feature selection: the new coordinate generally combines several original measurements.

PCA first centers the training data by subtracting each feature’s mean. Its first component is the direction along which the centered observations have the greatest variance. Each subsequent component captures the greatest remaining variance while being perpendicular to earlier components. In scikit-learn, components_ contains these directions; transform produces each observation’s coordinates along them. Keeping fewer directions reduces the number of columns without reducing the number of observations.

The phrase “preserves information” needs a qualification. PCA preserves variance, not necessarily useful predictive information. Among linear subspaces of a chosen dimension, ordinary PCA minimizes squared reconstruction error on the centered training data. A low-variance direction might nevertheless distinguish two important classes. Because PCA does not use target labels, it cannot know which variation matters for a later classification or regression task.

Scale determines what counts as large variation. Scikit-learn’s PCA centers features but does not standardize their variances. StandardScaler is often appropriate when features use different units or their numerical scales should not determine their influence. However, “always scale” is too absolute: when features share meaningful units and differences in variance deserve preservation, centering alone may be preferable. Standardization changes the problem and can amplify relatively uninformative, low-variance measurements.

The explained_variance_ratio_ values report the fraction of training variance captured by each retained component. Their sum describes retained variance, not model accuracy. Two components are convenient for plotting, but that convenience does not guarantee a faithful picture. PCA assumes a linear representation is useful; curved structure may require another method. Outliers can strongly affect its directions. Reducing dimensions may save downstream computation, although fitting PCA adds cost. Reconstruction restores the original feature layout, but discarded variation is generally unrecoverable, and interpreting mixed-feature components can be harder than interpreting the original measurements.

Practical walkthrough

Use an environment with Python 3.10 or later, NumPy, and scikit-learn already installed; this example uses APIs available in scikit-learn 1.2 and later. Everything runs in memory. The tiny dataset is deliberately constructed to expose PCA’s geometry, not to estimate performance on a realistic population.

Start by separating training observations from held-out observations. Here, X_train contains three paired readings, and X_test contains two readings reserved for transformation. The second feature has a numerical scale ten times larger than the first. We explicitly choose to give the two standardized features equal influence rather than let the larger-unit measurement dominate.

Build a Pipeline containing StandardScaler followed by PCA. Set n_components=1 to request one output column. The full singular-value-decomposition solver avoids randomized approximation and is sufficient for this small matrix. Call fit_transform only on X_train. This learns the scaler’s means and scales, then learns PCA’s directions from the scaled training observations.

Next, call transform on X_test. This reuses every learned quantity; it does not update the representation using held-out data. Inspect the output shapes and the retained training variance. Finally, use the pipeline’s inverse_transform to map the held-out scores back through PCA and undo standardization. Comparing these reconstructions with the original readings makes information loss visible in familiar units.

For a predictive project, place the downstream estimator after PCA in the same pipeline. Choose the component count using validation or cross-validation within the training data, then evaluate once on the untouched test set. Cross-validation must refit both preprocessing stages within each training fold to avoid leakage.

Worked example

import numpy as np
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

X_train = np.array([[1.0, 10.0], [2.0, 20.0], [3.0, 30.0]])
X_test = np.array([[4.0, 40.0], [4.0, 20.0]])

reducer = Pipeline([
    ("scale", StandardScaler()),
    ("pca", PCA(n_components=1, svd_solver="full")),
])

Z_train = reducer.fit_transform(X_train)
Z_test = reducer.transform(X_test)
reconstructed = reducer.inverse_transform(Z_test)
retained = reducer.named_steps["pca"].explained_variance_ratio_.sum()

print(f"Training shape: {X_train.shape} -> {Z_train.shape}")
print(f"Held-out shape: {X_test.shape} -> {Z_test.shape}")
print(f"Retained training variance: {retained:.1%}")
print("Held-out reconstructions:")
for row in reconstructed:
    print(f"{row[0]:.1f}, {row[1]:.1f}")

Expected output

Training shape: (3, 2) -> (3, 1)
Held-out shape: (2, 2) -> (2, 1)
Retained training variance: 100.0%
Held-out reconstructions:
4.0, 40.0
3.0, 30.0

All training rows satisfy second = 10 × first, so their standardized coordinates lie exactly on one line. One component therefore captures all training variance. The held-out reading (4, 40) follows that same relationship and reconstructs unchanged. The reading (4, 20) does not: projection discards its deviation from the learned line and reconstructs it as (3, 30). This is the closest point in standardized space, not necessarily in raw measurement units. Thus, complete training variance retention does not guarantee exact reconstruction of new observations. Formatting to one decimal place hides insignificant floating-point error; the output does not depend on component sign conventions.

Common mistakes and how to avoid them

• Fitting the scaler or PCA before splitting the dataset.

  Split first and fit both stages only on training observations. Unsupervised preprocessing can still leak information about held-out distributions; keep it inside the cross-validation pipeline.

• Interpreting 95% retained variance as 95% predictive accuracy.

  Treat the ratio as a geometric summary of the fitted data. Compare downstream validation performance with and without PCA; important target-related variation may be discarded.

• Standardizing automatically without considering measurement meaning.

  Inspect units and justify the desired weighting. Use standardization when equalized feature scales are appropriate, but preserve meaningful variance differences when the task calls for them.

• Treating inverse_transform as a way to recover deleted information.

  Compare reconstructed observations with their originals. Inversion returns a point on the retained subspace; it cannot restore an individual observation’s discarded component scores.

Review questions

Further reading

• scikit-learn user guide