Understanding Pipelines
A scikit-learn Pipeline represents an ordered learning procedure: transform the input features, then fit a final estimator. Its steps are named objects, such as a scaler followed by a classifier. Intermediate steps must support fitting and transformation; the final step must support fitting and may be a predictor or another transformer. The pipeline exposes methods supported by its final step, so a classification pipeline can usually be trained with fit and used with predict.
During fit(X_train, y_train), each intermediate transformer learns its state from the training data and transforms the features passed onward. StandardScaler, for example, learns a mean and standard deviation for each feature, then computes z = (x − mean) / standard deviation. The final estimator fits on the resulting representation. During predict(X_test), transformers reuse their fitted state; they do not learn new statistics from the test rows. The final estimator predicts from those transformed inputs.
This separation matters because preprocessing can learn information even without labels. If scaling, imputation, or feature selection uses held-out observations, those observations influence the training procedure. A pipeline helps prevent this leakage when all learned preprocessing is inside it and fitting receives only training data. It cannot repair target information embedded in a feature, duplicated observations across splits, or a split that ignores temporal ordering.
Cross-validation should therefore evaluate the entire pipeline. For each fold, scikit-learn fits a fresh copy on that fold’s training portion, including its preprocessing, and evaluates it on the validation portion. Passing only the classifier after globally transforming the dataset defeats this protection. Hyperparameter search follows the same principle: search over the complete procedure, not a model trained on an already contaminated representation.
Pipeline is sequential; ColumnTransformer handles different feature groups. It can scale numeric columns and encode categorical columns separately, concatenate their outputs, and feed that combined matrix to a model. These choices remain algorithm-dependent: scaling changes distance calculations and regularized models, but is usually unnecessary for decision trees. Pipelines improve consistency, not accuracy automatically. They also preserve a prediction-time recipe, but applications must still supply compatible columns, types, and feature meanings. Distribution shifts and unsuitable model assumptions remain problems outside the pipeline’s organizational guarantees.
Practical walkthrough
Start by defining what each row represents and which information would actually be available at prediction time. Separate the target from the features, then reserve test observations before fitting anything. The example below supplies training and held-out inputs explicitly so the training boundary is visible. It requires NumPy and scikit-learn, but no files or downloads.
Build a list of named steps. Here, scale applies StandardScaler and nearest applies a one-neighbor classifier. Nearest-neighbor prediction compares a new observation with stored training observations in transformed feature space. With one neighbor, its prediction is that neighbor’s label. This makes the mechanics easy to inspect without iterative optimization or random initialization.
Call fit once on the training inputs and labels. Inspect the learned scaler through named_steps to verify its mean. Then call predict on raw held-out inputs. Do not manually standardize them first: the pipeline already performs that operation. The separate transform call in the example is only for inspecting the intermediate representation.
For mixed data, replace the scaler with a ColumnTransformer containing numeric and categorical branches. A numeric branch might impute missing values before scaling; a categorical branch might impute before one-hot encoding. Explicitly choose how unseen categories should behave. Unselected columns are dropped by default, so check the column selection carefully.
For tuning, pass the pipeline to GridSearchCV and address nested parameters with names such as nearest__n_neighbors. set_params changes configuration; it does not perform a search or automatically retrain. Select settings using training-only cross-validation, refit, and evaluate once on the untouched test set.
Worked example
import numpy as np
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.neighbors import KNeighborsClassifier
# Rows are observations; the single column is a numeric feature.
X_train = np.array([[0.0], [2.0], [4.0], [6.0]])
y_train = np.array([0, 0, 1, 1])
X_test = np.array([[0.5], [5.5]])
model = Pipeline([
('scale', StandardScaler()),
('nearest', KNeighborsClassifier(n_neighbors=1)),
])
model.fit(X_train, y_train)
scaler = model.named_steps['scale']
print('Training mean:', scaler.mean_.round(3).tolist())
print('Training scale:', scaler.scale_.round(3).tolist())
print('Transformed test:', scaler.transform(X_test).round(3).tolist())
print('Predictions:', model.predict(X_test).tolist())
Expected output
Training mean: [3.0]
Training scale: [2.236]
Transformed test: [[-1.118], [1.118]]
Predictions: [0, 1]
The four training values have mean 3 and population variance 5, so StandardScaler uses a scale of √5. It transforms 0.5 into approximately −1.118 and 5.5 into approximately 1.118 using training statistics only. Their nearest training values are respectively 0 and 6, whose labels are 0 and 1. With one nonconstant feature, standardization multiplies all distances by the same positive factor, so it does not change neighbor ordering here. Multiple differently scaled features can change that ordering. These outputs demonstrate data flow, not predictive quality: two unlabeled held-out inputs cannot establish accuracy or justify choosing this model for real applications.
Common mistakes and how to avoid them
Scaling the whole dataset before cross-validation.
Place the scaler inside the pipeline passed to cross-validation. Each fold must learn its statistics without seeing its validation observations, even though scaling does not use labels.
Calling fit_transform on incoming prediction data.
Call the fitted pipeline’s predict method on raw features. Refitting preprocessing changes the coordinate system while the model still expects the training representation.
Assuming a pipeline makes every split valid.
Use group-aware splits for related observations or time-aware splits for forecasting. Remove features unavailable at prediction time; a pipeline cannot identify those semantic leaks.
Changing a nested setting and expecting updated predictions.
Use step__parameter syntax, then refit on training data. Treat preprocessing configuration changes as changes to the complete learning procedure, not merely prediction-time formatting.
Review questions
What does the scaler learn, and when is that knowledge reused?
It learns feature means and scales during fitting. Transformation and prediction reuse those stored values, including for observations whose distribution differs from the training data.
Why must cross-validation receive the whole pipeline?
Each validation fold must be excluded from every learned training operation. Evaluating the whole pipeline refits preprocessing and the model using only that fold’s training portion.
How do Pipeline and ColumnTransformer differ?
Pipeline passes outputs through successive steps. ColumnTransformer applies designated transformations to selected feature groups and combines their outputs; it commonly forms one preprocessing step inside a pipeline.
Does the example show that scaling improves nearest-neighbor accuracy?
No. In this single-feature example, scaling preserves neighbor ordering. There are also no held-out labels for measuring accuracy; the example demonstrates fitting and prediction mechanics only.
