Multiple Linear Regression

Understanding Multiple Linear Regression

Multiple linear regression predicts one numerical target from several input features. For a delivery example, the target might be journey time, with distance and number of stops as inputs. The model has the form predicted time = intercept + distance coefficient × distance + stops coefficient × stops. With two features, this describes a plane rather than the line used in simple regression. “Multiple” refers to the inputs, not to multiple prediction targets.

Training selects the intercept and coefficients together. Ordinary least squares, included in scikit-learn’s linear-model guidance, minimizes the sum of squared residuals on the training observations. A residual is the observed target minus its prediction. Squaring prevents positive and negative errors from cancelling, but also gives large errors disproportionate influence. Consequently, unusual observations can substantially change the fitted model; an apparently good average fit deserves inspection rather than automatic acceptance.

A coefficient describes a change in the model’s prediction when its feature increases by one unit and the other inputs remain fixed. If distance has coefficient 2, another kilometer adds two predicted minutes at the same stop count. This is a conditional association, not evidence that an intervention causes that change. The intercept is the prediction when every input equals zero, which may be outside the meaningful range of the data.

Coefficient magnitudes are not a universal ranking of feature importance. Kilometers and meters describe identical distances but produce differently sized coefficients. Feature variability and relationships between inputs also matter. Highly correlated inputs make it difficult to separate their contributions: predictions can remain useful while individual coefficients vary sharply across samples. Removing a feature is one option, not an automatic remedy; combining redundant measurements or validating a regularized model such as Ridge may be preferable.

Useful predictions require the additive relationship to approximate the problem and future observations to resemble the training setting. Curvature, interactions, missing predictors, and changing conditions can undermine this assumption. Normal errors are not required merely to fit or predict; classical uncertainty calculations need additional assumptions. Evaluate on untouched data, examine residual patterns, and avoid treating a small training error as proof of reliable future performance.

Practical walkthrough

Use Python 3.10 or newer with compatible NumPy and scikit-learn installations; this example uses APIs available in scikit-learn 1.2 and later. It needs no files, downloads, or random sampling. The observations are deliberately synthetic delivery records, not evidence about actual transportation. Their targets follow a known rule so that you can check the fitted coefficients by hand.

First, identify the column contract: every row contains distance in kilometers followed by stop count. X_train therefore has six rows and two columns, while y_train contains six corresponding journey times in minutes. The inputs vary separately, allowing the model to distinguish their contributions. If stop count were always exactly twice the distance, those contributions would not be uniquely identifiable.

Next, keep the two evaluation rows separate before fitting. Call fit only with X_train and y_train. LinearRegression includes an intercept by default, so do not add a column of ones. No preprocessing is necessary for these complete numerical inputs. In a larger exercise, learn imputation, scaling, or feature selection only from training data, preferably inside a pipeline; apply the learned transformations unchanged to evaluation inputs.

Finally, predict both held-out rows and calculate mean absolute error, the average absolute difference between predictions and targets. Its units remain minutes. Print coefficients with feature names rather than an unexplained array. Decimal formatting suppresses insignificant floating-point differences. The tiny evaluation set verifies the mechanics, not generalization: realistic assessment needs representative held-out observations, a baseline comparison, and a split that respects time or repeated entities when those structures occur.

Worked example

import numpy as np
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_absolute_error

# Columns: distance in kilometers, number of stops.
# Synthetic rule: minutes = 5 + 2 * distance + 3 * stops.
X_train = np.array([
    [1, 1],
    [1, 3],
    [3, 1],
    [3, 3],
    [2, 1],
    [2, 3],
], dtype=float)
y_train = np.array([10, 16, 14, 20, 12, 18], dtype=float)

X_test = np.array([[2, 2], [3, 2]], dtype=float)
y_test = np.array([15, 17], dtype=float)

model = LinearRegression()
model.fit(X_train, y_train)
predictions = model.predict(X_test)

print(f"Intercept: {model.intercept_:.1f} minutes")
print(f"Distance coefficient: {model.coef_[0]:.1f} minutes/km")
print(f"Stops coefficient: {model.coef_[1]:.1f} minutes/stop")
print("Predictions: " + ", ".join(f"{p:.1f}" for p in predictions))
print(f"Test MAE: {mean_absolute_error(y_test, predictions):.1f} minutes")

Expected output

Intercept: 5.0 minutes
Distance coefficient: 2.0 minutes/km
Stops coefficient: 3.0 minutes/stop
Predictions: 15.0, 17.0
Test MAE: 0.0 minutes

The model recovers the synthetic intercept and both coefficients because the training targets follow the stated additive rule without noise. For the first held-out delivery, it calculates 5 + 2 × 2 + 3 × 2 = 15 minutes. For the second, it calculates 5 + 2 × 3 + 3 × 2 = 17 minutes. Both predictions match their targets to the displayed precision, producing an MAE of 0.0 minutes. Floating-point arithmetic can leave tiny numerical residuals internally. This perfect displayed result demonstrates a controlled calculation; it does not suggest that real delivery times are deterministic or perfectly predictable.

Common mistakes and how to avoid them

• Calling the largest absolute coefficient the most important feature.

  Check units and observed variation first. Converting kilometers to meters divides the distance coefficient by 1,000 without changing predictions. Even standardized coefficients require caution when features are correlated.

• Changing feature order during prediction.

  Keep distance first and stops second in every input row. A reversed NumPy array still has the expected shape, so prediction can succeed while calculating the wrong quantity.

• Preprocessing all observations before the split.

  Split first, then fit every learned transformation on training observations only. Use a pipeline during cross-validation so each fold learns its own preprocessing without seeing validation information.

• Deleting correlated columns without checking the modeling goal.

  Inspect redundancy and coefficient stability, then compare justified alternatives using validation data. Ridge can stabilize estimation through shrinkage, trading some bias for reduced sensitivity; it does not establish causal interpretations.

Review questions

Further reading

• scikit-learn user guide