Linear Regression

Understanding Linear Regression

Linear regression predicts a numeric target from one or more measured features. For example, a delivery service might predict journey time from distance. Each row represents one journey, the feature records distance in kilometres, and the target records minutes. The prediction estimates a typical outcome at that distance, not a guarantee for an individual journey.

With one feature, the equation is ŷ = b₀ + b₁x. Here, ŷ is the predicted target, b₀ is the intercept, and b₁ is the slope. A slope of 1.4 minutes per kilometre means that increasing distance by one kilometre increases the model's prediction by 1.4 minutes. The intercept is the prediction at zero distance; it may lack practical meaning if zero lies outside the observed data.

With several features, the model adds one coefficient times each feature. Each coefficient describes a change in the prediction while the other included features remain fixed. This is a conditional association, not proof of causation. Correlated features, such as distance and fuel consumption, can make individual coefficients unstable even when predictions remain useful. Coefficient sizes also depend on measurement units, so larger numbers do not automatically indicate greater importance.

Ordinary least squares chooses coefficients that minimise the sum of squared training residuals. A residual is observed target minus prediction. Squaring prevents positive and negative errors from cancelling and gives large errors disproportionate influence. This makes the method easy to interpret but sensitive to unusual observations. Minimising training error does not establish that the model will predict new observations accurately.

The central modelling assumption is that the target's conditional mean is adequately represented by the chosen linear expression. Independent observations matter for standard uncertainty calculations, and constant error variance supports conventional standard errors. Normally distributed errors are not required simply to fit a line or produce predictions. Residual curvature suggests missing structure; widening residual spread suggests changing uncertainty, not necessarily an unusable mean prediction. Linear regression can include squared features while remaining linear in its coefficients. Its simplicity is valuable, but extrapolation beyond observed conditions, omitted predictors, and changing populations can undermine predictions. Evaluation must therefore consider both numerical errors and the setting where predictions will be used.

Practical walkthrough

Begin by specifying the prediction task and its timing. For delivery duration, distance must be available before the journey finishes. A feature recorded only after arrival would leak information unavailable when making a real prediction. Decide whether an error of several minutes is acceptable before choosing a metric.

Inspect the table's units, missing values, duplicate records, and target definition. Pandas provides tools for these checks, but cleaning decisions still require domain knowledge. Do not automatically remove an unusually long journey: it might be a recording mistake or a genuine event the model must handle.

Separate training and evaluation observations before learning preprocessing rules. For future journeys, a chronological split may be more realistic than shuffling. If multiple records describe the same driver or route, consider whether the split should keep groups separate. Fit imputers, encoders, and scalers using training data only; reuse those fitted transformations on evaluation data. Scikit-learn pipelines help enforce this sequence. Scaling is not mandatory for ordinary least squares.

Fit the regression and compare its predictions with a simple baseline, such as always predicting the training target mean. Report mean absolute error in target units. Root mean squared error also uses target units but penalises large misses more strongly. R² measures improvement over a reference based on the evaluated targets' mean and can be negative.

Finally, examine signed residuals across distances and relevant groups. Use validation data for revisions, reserving a final test set for assessment. The tiny fixed split below illustrates arithmetic, not reliable evidence about real delivery performance.

Worked example

from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Invented journeys: distance (km) and duration (minutes).
X_train = [[1.0], [2.0], [3.0], [4.0]]
y_train = [3.0, 4.0, 6.0, 7.0]
X_test = [[1.5], [3.5]]
y_test = [4.0, 7.0]

model = LinearRegression().fit(X_train, y_train)
predictions = model.predict(X_test)
residuals = [actual - predicted
             for actual, predicted in zip(y_test, predictions)]
baseline = [sum(y_train) / len(y_train)] * len(y_test)

print(f"Slope: {model.coef_[0]:.2f} min/km")
print(f"Intercept: {model.intercept_:.2f} min")
print("Predictions:", ", ".join(f"{v:.2f}" for v in predictions))
print("Residuals:", ", ".join(f"{v:.2f}" for v in residuals))
print(f"Model MAE: {mean_absolute_error(y_test, predictions):.2f} min")
print(f"Baseline MAE: {mean_absolute_error(y_test, baseline):.2f} min")

Expected output

Slope: 1.40 min/km
Intercept: 1.50 min
Predictions: 3.60, 6.40
Residuals: 0.40, 0.60
Model MAE: 0.50 min
Baseline MAE: 1.50 min

This example requires scikit-learn and uses no external data. Feature inputs are two-dimensional: each inner list represents one journey with one feature. Training produces the equation predicted minutes = 1.50 + 1.40 × kilometres. At 1.5 kilometres, the prediction is 3.60 minutes; the observed value is 4.00, giving a positive residual of 0.40. Both held-out journeys are underpredicted. Their absolute errors average 0.50 minutes, compared with 1.50 for the training-mean baseline. Both distances lie inside the training range, so these are interpolation examples. Two evaluation observations cannot establish general accuracy, and the fitted coefficients do not describe a causal mechanism.

Common mistakes and how to avoid them

• Interpreting a coefficient as a causal effect.

  Describe how the fitted prediction changes, holding included features fixed. Investigate confounding and study design before claiming that changing a feature would change the outcome.

• Learning preprocessing rules from the entire dataset.

  Split first. Fit missing-value replacements and other learned transformations on training data only, ideally inside a pipeline that is refitted within each cross-validation fold.

• Treating every residual pattern as proof that linear regression must be abandoned.

  Diagnose the pattern. Curvature may justify transformed features; widening spread may require different uncertainty estimates. Compare proposed changes on validation data.

• Trusting a low training error or distant extrapolation.

  Evaluate unseen observations against a baseline and check whether new inputs resemble training conditions. Flag predictions outside the observed range rather than presenting them as equally trustworthy.

Review questions

Further reading

• pandas user guide
• scikit-learn user guide