Understanding Overfitting & Underfitting
Overfitting and underfitting describe how well a model learns patterns that remain useful beyond its training observations. Training error measures performance on examples used to fit the model. Validation error measures performance on separate examples used to compare modeling choices. Generalisation means predicting well on relevant unseen observations, not merely reproducing recorded outcomes.
Underfitting occurs when a fitted model misses useful structure. For example, predicting the same delivery time for every journey ignores a relationship with distance. Training and validation errors may both be high. Excessive restrictions, unsuitable features, or incomplete optimisation can produce this pattern. However, poor performance alone does not prove underfitting: unreliable measurements and unavailable predictors can also limit achievable accuracy.
Overfitting occurs when fitting captures sample-specific variation that does not generalise. A flexible curve might follow every measurement fluctuation while predicting poorly between observations. Literal memorisation is one possibility, not the definition. Typically, training error is substantially lower than validation error. That gap is evidence to investigate, rather than a diagnosis by itself: different populations, changing conditions, and small validation samples can also create gaps.
Judge errors against a meaningful baseline and the decision being supported. For regression, mean squared error averages squared prediction errors; lower is better, but its units are squared target units. Mean absolute error is often easier to interpret operationally. For classification, accuracy can conceal failure on rare outcomes. Neither a small gap nor high training accuracy establishes that a model is useful.
Model flexibility creates a trade-off. More flexibility can capture genuine relationships but can also follow noise. Regularisation discourages complexity, potentially improving validation performance while worsening training fit. Too much regularisation can cause underfitting. More representative training data may help an overfitting model, but duplicate records or irrelevant features are not equivalent to additional information.
These comparisons assume evaluation data reasonably represent intended use and remain independent of fitting. Predictive associations do not establish causation. The scikit-learn guide separates metrics, cross-validation, validation curves, and leakage prevention because each addresses a different evaluation question. Together, these tools support diagnosis; none supplies a universal error threshold or guarantees reliable performance after deployment, especially when future conditions differ.
Practical walkthrough
Start by defining the prediction task: what outcome is predicted, when the prediction is made, and which information exists at that moment. Choose an error measure that reflects the cost of mistakes. Establish a simple baseline, such as predicting the training-set mean for a regression task.
Separate data before fitting anything. Use training data to estimate model parameters, validation data to choose settings, and an untouched test set for final evaluation. For future-event prediction, preserve time order. For repeated observations from customers or devices, keep related records together when evaluating generalisation to new entities.
Fit imputation, scaling, and feature selection on training data only. Apply the resulting transformations unchanged to validation data. During cross-validation, repeat fitting inside each training fold; a pipeline helps keep these operations together. Cross-validation compares choices across several splits, but its splitting strategy must still match the intended use.
Compare a small, planned range of model complexities using the same metric and validation observations. Record training and validation errors together. If both are poor, investigate features, optimisation, and model restrictions. If training improves while validation worsens, consider stronger regularisation, fewer features, or reduced complexity. Inspect errors across relevant subgroups as well as overall.
Use learning curves to examine errors as training size increases; use validation curves to examine errors as a setting changes. Neither requires plotting: a table can suffice. Select the modeling procedure using validation results, then evaluate it once on the test set. Report limitations, including sample size and whether evaluation covers the conditions expected in use.
Worked example
# Pure Python: no packages or downloads required.
# Fixed artificial observations around the relationship y = x.
x_train = [-3, -1, 1, 3]
y_train = [-2, -2, 2, 2]
x_valid = [-2, 0, 2]
y_valid = [-2, 0, 2]
x_mean = sum(x_train) / len(x_train)
y_mean = sum(y_train) / len(y_train)
slope = sum((x - x_mean) * (y - y_mean)
for x, y in zip(x_train, y_train)) / sum(
(x - x_mean) ** 2 for x in x_train)
def constant(x):
return y_mean
def linear(x):
return y_mean + slope * (x - x_mean)
def cubic(x):
# Lagrange interpolation: the cubic through all training points.
total = 0.0
for i, xi in enumerate(x_train):
term = float(y_train[i])
for j, xj in enumerate(x_train):
if i != j:
term *= (x - xj) / (xi - xj)
total += term
return total
def mse(model, xs, ys):
return sum((model(x) - y) ** 2
for x, y in zip(xs, ys)) / len(xs)
print('model train_MSE validation_MSE')
for name, model in [('constant', constant),
('linear', linear), ('cubic', cubic)]:
print(f'{name} {mse(model, x_train, y_train):.3f} '
f'{mse(model, x_valid, y_valid):.3f}')
Expected output
model train_MSE validation_MSE
constant 4.000 2.667
linear 0.800 0.107
cubic 0.000 0.667
The constant model ignores the relationship and has the largest errors. The fitted line has slope 0.8 and accepts some training error, yet predicts the validation observations most accurately. The cubic passes through every training point, reducing training error to zero, but its validation error exceeds the line's. This illustrates overfitting relative to the linear alternative, not a rule that cubic models are always inappropriate. All models use only training observations for fitting. The validation values intentionally follow the underlying artificial relationship exactly; real measurements usually contain noise. These tiny samples demonstrate a mechanism, not a trustworthy final performance estimate.
Common mistakes and how to avoid them
Treating any training–validation gap as proof of overfitting.
Check sample sizes, group overlap, time periods, and target distributions. Compare several suitable validation splits before deciding that model flexibility is the main problem.
Choosing the model with the smallest gap.
Prioritise validation performance against a baseline. A constant predictor can have similarly poor errors on both sets; matching errors do not imply useful predictions.
Preprocessing all observations before splitting.
Split first. Learn scaling statistics, imputation values, and selected features from training observations only, including within each cross-validation fold. Otherwise, held-out information influences fitting.
Repeatedly using test results to adjust the model.
Tune on validation data or cross-validation instead. Once test results guide changes, that set is no longer an independent final check; obtain fresh evaluation data.
Review questions
Why can zero training error be undesirable?
It may reflect fitting measurement noise or sample-specific details. In the example, the cubic fits perfectly but predicts validation observations less accurately than the line.
What should you investigate when both errors are high?
Compare with a baseline, inspect data quality and available features, and verify fitting completed properly. Then test whether relaxing model restrictions improves validation performance.
How can regularisation help, and what is its limitation?
It discourages overly complex fitted relationships, potentially reducing sensitivity to training noise. Excessive regularisation suppresses useful structure and can worsen both training and validation errors.
What can a learning curve tell you about collecting more data?
Improving validation error as training size grows suggests additional representative observations may help. Persistently poor, converged errors suggest investigating features or model restrictions, though neither pattern guarantees future results.
