Understanding Regularization (L1 / L2)
Regularization changes what a learning algorithm considers a good fit. Ordinary least squares chooses coefficients that minimize squared prediction errors on training observations. A regularized model also pays a penalty for its coefficients. It may accept larger training errors in exchange for a less sensitive fitted model. This can improve predictions on unseen observations, but improvement is not guaranteed: excessive regularization produces underfitting.
For linear regression, predictions are b + Xw, where b is the intercept and w contains feature coefficients. Let SSE denote the sum of squared residuals and n the number of training observations. Scikit-learn’s Ridge minimizes SSE + alpha × sum(w_j²). Its Lasso minimizes SSE/(2n) + alpha × sum(|w_j|). These different normalizations matter: identical numerical alpha values do not represent equivalent penalty strengths across these estimators. The intercept is ordinarily unpenalized.
L2 uses a smooth quadratic penalty. Its derivative grows with coefficient magnitude, making large coefficients increasingly expensive. For centered data, Ridge solves (XᵀX + alpha I)w = Xᵀy. Adding a positive diagonal term stabilizes estimation when features are strongly correlated. Ridge generally retains nonzero coefficients rather than selecting a subset. However, “shrinks all coefficients” is shorthand: with correlated predictors, an individual coefficient need not decrease monotonically as alpha increases.
L1 uses absolute values, creating a corner at zero. In coordinate descent, Lasso repeatedly updates one coefficient while holding the others fixed. An update applies soft thresholding: a sufficiently small relationship between a feature and the remaining residual becomes exactly zero. This explains sparsity mechanically, rather than treating feature selection as a separate procedure. Correlated features can substitute for one another, so the selected subset may be unstable.
Penalties depend on feature units. Rescaling a feature upward allows a smaller coefficient for the same prediction, reducing its penalty. Standardization usually makes penalization more comparable, although scaling choices should reflect the application. Larger alpha means stronger regularization within either estimator. Elastic Net combines L1 and L2 penalties, trading sparsity against stabilization. None of these methods establishes causation, repairs missing predictive information, or guarantees that a linear relationship is appropriate. Validation must determine whether the resulting trade-off is useful.
Practical walkthrough
Begin by defining the prediction task, available features, and evaluation metric. Reserve a test set before fitting anything. Use splits that reflect deployment: chronological splits for forecasting, or group-aware splits when observations share a person or organization. Otherwise, validation can reward memorizing relationships that will not transfer to genuinely new cases.
Build a pipeline containing StandardScaler followed by Ridge or Lasso. During cross-validation, fit the entire pipeline separately inside each training fold. The scaler must learn means and standard deviations only from that fold; validation observations receive the already learned transformation. Scaling once before cross-validation leaks information, even though scaling does not use the target.
Compare several positive alpha values spanning orders of magnitude, then refine the range if needed. Select alpha using validation error, not training error or the number of zero coefficients alone. Search Ridge and Lasso independently because their objective normalizations differ. Include an unregularized baseline when appropriate. If Lasso reports convergence warnings, investigate scaling, increase the iteration budget, and check solver tolerance before interpreting its coefficients.
After selection, refit the chosen pipeline on all development data and evaluate once on the untouched test set. Inspect both prediction error and coefficient stability. A sparse model can be convenient, but convenience is not evidence that discarded features are irrelevant.
The example below isolates the mechanics instead of estimating generalization. Its four constructed observations have centered, orthogonal, unit-variance features, so standardization leaves them unchanged. Fixed alpha values make shrinkage easy to verify analytically; they are demonstration settings, not tuned recommendations for another dataset.
Worked example
import numpy as np
from sklearn.linear_model import LinearRegression, Ridge, Lasso
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import StandardScaler
# Constructed training data: centered, orthogonal columns.
X_train = np.array([
[-1.0, -1.0],
[-1.0, 1.0],
[ 1.0, -1.0],
[ 1.0, 1.0],
])
y_train = 2.0 * X_train[:, 0] + 0.25 * X_train[:, 1]
models = [
("OLS", LinearRegression()),
("Ridge", Ridge(alpha=4.0, solver="cholesky")),
("Lasso", Lasso(alpha=0.5, tol=1e-12, max_iter=10000)),
]
for name, estimator in models:
pipeline = make_pipeline(StandardScaler(), estimator)
pipeline.fit(X_train, y_train)
coefficients = pipeline.steps[-1][1].coef_
text = ", ".join(f"{value:.3f}" for value in coefficients)
print(f"{name}: [{text}]")
Expected output
OLS: [2.000, 0.250]
Ridge: [1.000, 0.125]
Lasso: [1.500, 0.000]
This example requires NumPy and scikit-learn. Ordinary least squares recovers the constructed coefficients exactly. Because XᵀX equals 4I, Ridge multiplies each ordinary least-squares coefficient by 4/(4 + alpha). With alpha equal to four, both coefficients halve. For these orthogonal, unit-variance columns, Lasso instead soft-thresholds each ordinary least-squares coefficient by alpha: 2 becomes 1.5, while 0.25 becomes zero. This simple rule does not directly apply to correlated columns. The scaler is fitted only on training data. No test score is reported because this tiny, noiseless construction demonstrates optimization behavior, not evidence that either penalty improves generalization.
Common mistakes and how to avoid them
Assuming large-scale features receive stronger penalties.
For an equivalent prediction contribution, larger feature values require smaller coefficients and therefore smaller penalties. Standardize suitable numeric predictors within the training pipeline.
Choosing alpha from the final test results.
Choose it through validation or cross-validation on development data. Keep the final test set untouched until all model choices are fixed.
Treating a zero Lasso coefficient as proof of irrelevance.
Check stability across validation folds and consider correlated substitutes. Zero means exclusion under this fitted objective, not absence of predictive or causal relevance.
Expecting stronger regularization always to improve predictions.
Compare validation errors across strengths. Excessive shrinkage removes useful predictive signal; select the trade-off empirically rather than maximizing sparsity.
Review questions
Why does Lasso produce exact zeros more readily than Ridge?
The absolute-value penalty has a corner at zero, allowing soft-thresholded updates to stop there. Ridge’s smooth quadratic penalty usually reduces coefficients without making them exactly zero.
Why can regularization increase training error yet help on new data?
It restricts how closely coefficients adapt to training-specific variation. That introduces bias but can reduce estimation variance. Whether the balance improves unseen predictions must be checked with validation.
Can Ridge(alpha=1) and Lasso(alpha=1) be considered equally regularized?
No. Their penalties have different shapes, and scikit-learn normalizes their squared-error terms differently. Tune each estimator independently rather than interpreting equal alpha values as equal strength.
Where should standardization occur during cross-validation?
Inside the pipeline fitted on each training fold. Apply that fold’s learned transformation to its validation fold without refitting, preventing validation information from influencing preprocessing.
