Polynomial Regression

Understanding Polynomial Regression

Polynomial regression changes the inputs to a linear model rather than replacing its fitting method. Suppose a sensor’s response bends upward as its input increases. A straight line, ŷ = b₀ + b₁x, has one constant slope and cannot represent that bend. A quadratic model, ŷ = b₀ + b₁x + b₂x², adds a feature whose contribution grows with the square of the input. The coefficients determine the resulting curve.

The word “linear” describes how the unknown coefficients enter the equation. Each coefficient multiplies a known feature, and these contributions are added. Neither squaring x nor including x³ makes the model nonlinear in its coefficients. Consequently, ordinary least squares can estimate polynomial coefficients using the same machinery used for straight-line regression: it minimizes the sum of squared differences between predictions and observed targets.

In scikit-learn, PolynomialFeatures constructs these additional columns, while LinearRegression estimates their weights. For one input and degree two, include_bias=False produces columns x and x². LinearRegression supplies the intercept separately by default. With two inputs, a and b, degree two produces a, b, a², ab, and b². Thus, polynomial expansion introduces interactions as well as powers; the number of columns can grow quickly with both degree and input count.

Degree controls flexibility, not guaranteed quality. A low degree may miss genuine curvature, while a high degree may follow noise or become numerically unstable. Higher degree increases overfitting risk; it does not prove that overfitting occurred. Compare candidate degrees using validation data or cross-validation, keeping the final test set untouched until selection is complete. Training error alone rewards unnecessary flexibility.

The practical assumption is that a polynomial approximates the predictive relationship over the region of interest. Smoothness alone does not guarantee a useful approximation at a chosen degree. Predictions outside the observed input range can grow sharply and become implausible. Correlated power columns also make individual coefficients sensitive to small data changes. Scaling and regularization can help, but neither guarantees reliable extrapolation. These limitations matter even when the training curve looks excellent. A fitted association, moreover, does not establish that changing the input causes the predicted response.

Practical walkthrough

Use Python 3.10 or later with compatible installations of NumPy and scikit-learn; this example uses APIs available in scikit-learn 1.2 and later. Everything runs in memory. No dataset download, plotting library, or random seed is needed. Familiarity with arrays and ordinary linear regression is sufficient.

First, create five training inputs from −2 through 2. The targets follow the deliberately chosen rule y = 1 + 2x + 3x². This artificial setup isolates feature expansion from measurement noise. Reshape the inputs into five rows and one column: scikit-learn expects a two-dimensional feature matrix even when there is only one original feature.

Next, reserve two separate inputs, −1.5 and 1.5, for evaluation. They fall within the training range but are not training observations. Degree two is specified because the demonstration’s generating rule is known, not because the evaluation results were inspected. Real applications generally require degree selection using training-only cross-validation.

Build a Pipeline containing PolynomialFeatures and LinearRegression. Calling fit on the pipeline fits its steps using training data only. PolynomialFeatures does not estimate means or variances, but keeping it inside the pipeline preserves consistent transformations and makes later additions, such as scaling, safer. During prediction, the existing transformation is reused rather than refitted.

Finally, inspect the generated feature names, fitted parameters, predictions, and mean absolute error. Formatting numbers to two decimal places hides insignificant floating-point differences. Scaling is omitted because this small, low-degree example has modest input magnitudes. For larger powers, consider scaling and Ridge regression inside the pipeline, selecting their settings without consulting final test results.

Worked example

import numpy as np
from sklearn.preprocessing import PolynomialFeatures
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import Pipeline
from sklearn.metrics import mean_absolute_error

X_train = np.array([-2., -1., 0., 1., 2.]).reshape(-1, 1)
y_train = 1 + 2 * X_train[:, 0] + 3 * X_train[:, 0] ** 2

X_test = np.array([-1.5, 1.5]).reshape(-1, 1)
y_test = 1 + 2 * X_test[:, 0] + 3 * X_test[:, 0] ** 2

model = Pipeline([
    ("powers", PolynomialFeatures(degree=2, include_bias=False)),
    ("regression", LinearRegression()),
])
model.fit(X_train, y_train)
predictions = model.predict(X_test)

powers = model.named_steps["powers"]
regression = model.named_steps["regression"]
print("Features:", ", ".join(powers.get_feature_names_out(["x"])))
print(f"Intercept: {regression.intercept_:.2f}")
print("Coefficients:", ", ".join(f"{v:.2f}" for v in regression.coef_))
print("Predictions:", ", ".join(f"{v:.2f}" for v in predictions))
print(f"Test MAE: {mean_absolute_error(y_test, predictions):.2f}")

Expected output

Features: x, x^2
Intercept: 1.00
Coefficients: 2.00, 3.00
Predictions: 4.75, 10.75
Test MAE: 0.00

The expansion gives the estimator two columns while retaining five training rows. Their fitted coefficients recover the constructed rule: an intercept of 1, an x coefficient of 2, and an x² coefficient of 3. At x = −1.5, the prediction is 1 − 3 + 6.75 = 4.75; at x = 1.5, it is 1 + 3 + 6.75 = 10.75. The displayed mean absolute error is zero because the noiseless quadratic lies within the model’s representational capacity. Tiny numerical residuals may exist before formatting. This illustrates correct mechanics, not evidence that quadratic models generally predict real measurements without error.

Common mistakes and how to avoid them

• Choosing the degree with the smallest training error.

  Compare a small range of degrees using training-only cross-validation. Prefer a simpler model when validation performance is comparable, and evaluate the selected pipeline once on untouched test data.

• Creating a constant feature while also fitting an intercept.

  Set include_bias=False when LinearRegression uses its default intercept. Otherwise, the constant column duplicates the intercept’s role and makes the parameterization redundant.

• Fitting a scaler on all observations before splitting.

  Split first and put scaling inside the pipeline. Cross-validation must refit learned preprocessing on each training fold, not reuse statistics computed from its validation fold.

• Treating an accurate in-range curve as trustworthy everywhere.

  Check new inputs against the training range. Flag extrapolation and assess whether the curve’s growth is plausible; low test error within the range does not validate distant predictions.

Review questions

Further reading

• scikit-learn user guide