Linear Regression

Understanding Linear Regression

Linear regression predicts a numerical target from one or more measured features. Imagine estimating packing time from the number of items in an order. With one feature, the prediction is ŷ = b + wx: x is the item count, w is the slope, and b is the intercept. The hat distinguishes a prediction from an observed target y. Fitting means choosing b and w from labeled training examples, rather than specifying them manually.

Ordinary least squares, the approach implemented by scikit-learn’s LinearRegression, chooses coefficients that minimize the sum of squared training residuals. A residual is y − ŷ, the observed value minus its prediction. Squaring prevents positive and negative residuals from canceling and penalizes large misses disproportionately. Consequently, an unusual observation can pull the fitted line toward itself. Minimizing this training objective does not guarantee accurate predictions for new orders.

With several features, the equation becomes ŷ = b + w₁x₁ + w₂x₂ + … . Each coefficient describes the prediction change for a one-unit feature increase while other features remain fixed. Its size depends on measurement units, so coefficient magnitudes are not automatically comparable measures of importance. Correlated features can make individual coefficients unstable. Neither a coefficient nor a good predictive score establishes that changing a feature causes the target to change.

A straight-line model assumes the conditional average target is adequately represented by the selected linear combination. It need not pass through every observation. Independent observations and stable relationships between training and future data support meaningful evaluation. Constant error variance and normally distributed errors matter for classical uncertainty calculations, but are not prerequisites for calling fit or predict. LinearRegression alone does not provide prediction intervals.

The supplied scikit-learn guide places ordinary least squares alongside polynomial features, regularization, evaluation, and leakage prevention. These connections matter: a curved relationship does not make all linear models useless. Adding x² can represent curvature while remaining linear in the learned coefficients. Extra flexibility can overfit, however. Begin with a simple baseline, examine held-out errors, and avoid trusting extrapolation far beyond observed feature values. Plots can reveal curvature, outliers, or changing error spread that one summary score hides.

Practical walkthrough

Use Python 3.10 or later with compatible NumPy and scikit-learn installations; this example uses APIs available in scikit-learn 1.2 and later. You should understand lists, imports, and basic arithmetic. All data are created in memory, and no files or external services are involved.

First, represent each order as one row of X. Even with one feature, scikit-learn expects a two-dimensional feature matrix: four orders with one feature have shape (4, 1). The target y is a one-dimensional sequence containing four packing times in minutes. Row order must match between X and y.

Second, reserve two separate orders for evaluation before fitting. Here, fixed synthetic training observations follow time = 3 + 2 × items, while the held-out observations deliberately differ slightly. This setup makes every calculation inspectable; it is not evidence about an actual packing operation. Both test item counts lie inside the training range, so this exercise evaluates interpolation rather than extrapolation.

Third, call fit using only training data, then inspect coef_ and intercept_. No scaling or imputation is needed here. If you add learned preprocessing later, fit it only on training data and apply those fitted transformations unchanged to test data, preferably through a pipeline.

Finally, predict held-out times and calculate mean absolute error, or MAE. Unlike the squared-error fitting objective, MAE reports the average absolute miss in the target’s original units. Compare it with a baseline that always predicts the training mean. Two test orders demonstrate the workflow, but cannot establish reliable performance; realistic evaluation needs more representative held-out observations.

Worked example

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Synthetic orders: item count -> packing time in minutes.
X_train = np.array([[1.0], [3.0], [5.0], [7.0]])
y_train = np.array([5.0, 9.0, 13.0, 17.0])
X_test = np.array([[2.0], [6.0]])
y_test = np.array([8.0, 14.0])

model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)
baseline = np.full(y_test.shape, y_train.mean())

print(f"Slope: {model.coef_[0]:.1f}")
print(f"Intercept: {model.intercept_:.1f}")
print("Predictions:", ", ".join(f"{p:.1f}" for p in predictions))
print("Residuals:", ", ".join(f"{r:.1f}" for r in y_test - predictions))
print(f"Model MAE: {mean_absolute_error(y_test, predictions):.1f}")
print(f"Baseline MAE: {mean_absolute_error(y_test, baseline):.1f}")

Expected output

Slope: 2.0
Intercept: 3.0
Predictions: 7.0, 15.0
Residuals: 1.0, -1.0
Model MAE: 1.0
Baseline MAE: 3.0

The training points lie exactly on ŷ = 3 + 2x, so fitting recovers a slope of two minutes per item and an intercept of three minutes. Substituting two and six items gives seven and fifteen minutes. Observed test times are eight and fourteen, producing residuals of positive one and negative one. Their absolute values average to one minute. The training mean is eleven minutes; predicting eleven for both test orders gives absolute errors of three minutes each. Formatting to one decimal place suppresses insignificant floating-point differences. The intercept is an extrapolated prediction at zero items, not an observed setup time.

Common mistakes and how to avoid them

• Passing a flat feature list because there is only one input variable.

  Use one nested row per observation, such as [[2.0], [6.0]]. Keep the same feature order and number of columns during training and prediction.

• Reporting training error as evidence of future accuracy.

  Evaluate on untouched observations. Keep preprocessing and model selection away from the final test set; for time-ordered data, train on earlier observations rather than leaking future information.

• Treating a coefficient as a causal effect or an unrestricted prediction rule.

  Interpret it as a model association conditional on included features. Check units and observed ranges, and investigate correlated inputs before making substantive claims.

• Automatically deleting outliers to improve the score.

  Check whether unusual values are recording errors or valid cases. Correct verified errors; for genuine extremes, examine sensitivity and consider robust regression rather than silently removing difficult observations.

Review questions

Further reading

• scikit-learn user guide