Dimensionality Reduction (PCA)

Understanding Dimensionality Reduction (PCA)

Principal component analysis (PCA) replaces numeric features with new coordinates called principal component scores. Each component is a weighted combination of the original features, after centering and any chosen scaling. Instead of selecting existing columns, PCA constructs new axes. Reducing dimensionality means retaining only some of these axes, so each observation needs fewer numbers to describe it.

Imagine two measurements that usually increase together. A scatterplot may form an elongated cloud along a diagonal. The first principal component follows the direction with the greatest variance: the largest spread of projected observations. The second follows the greatest remaining variance while being perpendicular to the first. Further components follow the same rule. Their scores are uncorrelated on the fitted data, but not necessarily statistically independent.

The explained variance ratio measures the fraction of total input variance associated with each component. Ratios of 0.72 and 0.18 would mean that two components retain 90% of that variance. This is not 90% predictive accuracy, nor necessarily 90% of useful information. PCA does not inspect a prediction target. A low-variance direction could contain an important signal, while a high-variance direction could reflect measurement noise or an outlier.

Scale therefore changes the question PCA answers. Scikit-learn's PCA centers features but does not automatically standardize them. Standardization gives each nonconstant feature unit variance before PCA, preventing units such as dollars from dominating units such as years merely through numerical magnitude. However, scaling is a choice, not an absolute rule. When features share meaningful units and their variance differences matter, unscaled PCA may be appropriate. Standardization can also amplify a noisy feature with little original variation.

For interpretation, distinguish component weights from observation scores. Weights describe an axis; scores locate observations along it. Large absolute weights identify features contributing strongly to that axis under the chosen preprocessing. They do not establish causes. PCA captures linear structure and can miss curved patterns. Dropping components can simplify visualization or suppress some noise, but it also discards differences between observations. The practical question is whether those discarded differences matter for the analysis, not simply whether the retained percentage looks impressive.

Practical walkthrough

Start by defining what one row represents and which measurements belong in the analysis. Select numeric features deliberately; identifiers and numeric category codes rarely represent meaningful distances. Inspect missing values, constant columns, unusual observations, and units. The pandas guide's selection, missing-data, and descriptive-statistics topics support this preparation. Ordinary PCA requires an appropriate numeric matrix, not an untreated mixed-type table.

For predictive work, split training and test observations before learning transformations. Fit imputation, scaling, and PCA using training data only. Apply those fitted transformations to validation or test data without refitting. A scikit-learn Pipeline keeps these steps together; during cross-validation, fit the entire pipeline separately within each training fold. Unsupervised preprocessing can still leak information about held-out observations.

Choose scaling according to the measurement context. Then inspect explained variance ratios and their cumulative sum. For a two-dimensional display, two components are convenient, but report how much variance they omit. For compression, a cumulative variance threshold can guide selection. For prediction, compare candidate component counts through validation and include a baseline without PCA. Fewer columns do not guarantee better predictions or lower total runtime.

The example uses four training observations with two equally scaled measurements. Their strongest variation runs along the diagonal where both measurements rise together. Keeping one component preserves that shared variation and removes the contrast between measurements. A separate observation is transformed and reconstructed using only the training fit. Reconstruction makes the loss visible in original units: two distinct measurements become equal. This is an illustration of compression, not evidence of denoising success.

Worked example

import numpy as np
from sklearn.decomposition import PCA
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Requires NumPy and scikit-learn; no external data.
X_train = np.array([
    [2.0, 2.0],
    [-2.0, -2.0],
    [1.0, -1.0],
    [-1.0, 1.0],
])
X_new = np.array([[3.0, 1.0]])

reducer = Pipeline([
    ('scale', StandardScaler()),
    ('pca', PCA(n_components=1, svd_solver='full')),
])
train_scores = reducer.fit_transform(X_train)
new_scores = reducer.transform(X_new)
reconstructed = reducer.inverse_transform(new_scores)
ratio = reducer.named_steps['pca'].explained_variance_ratio_[0]

print('Training score shape:', train_scores.shape)
print(f'Retained variance: {ratio:.3f}')
print(f'Discarded variance: {1 - ratio:.3f}')
print('Reconstructed new row: '
      f'[{reconstructed[0, 0]:.3f}, {reconstructed[0, 1]:.3f}]')

Expected output

Training score shape: (4, 1)
Retained variance: 0.800
Discarded variance: 0.200
Reconstructed new row: [2.000, 2.000]

Both training columns have mean zero and equal variance, so standardization preserves their relative geometry. The diagonal direction contains four times as much variance as the perpendicular contrast direction; retaining it therefore keeps 80% of training variance. The score matrix has four rows but only one column. Inverse transformation maps the new observation back through PCA and scaling. Its reconstruction, [2, 2], retains the shared level of [3, 1] but loses their difference. Component signs can reverse without changing this reconstruction. The printed variance ratio describes the training distribution, not a guarantee about variance retained in future observations.

Common mistakes and how to avoid them

• Treating retained variance as retained predictive information.

  Report it specifically as input variance under the chosen preprocessing. Evaluate predictive usefulness separately on validation data, including a comparison without PCA.

• Always standardizing without considering measurement meaning.

  Check units, noise, and whether variance differences are meaningful. Document the scaling choice; inspect how it changes component weights and the retained variance.

• Fitting PCA or scaling before the train-test split.

  Split first, then fit all learned preprocessing on training observations. Use a pipeline inside cross-validation so each fold learns its own transformations.

• Reading a two-component view as the complete data structure.

  Report the omitted variance and inspect additional components when necessary. Apparent overlap can hide separation in discarded directions; apparent groups require further investigation.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide