Overfitting & Underfitting

Understanding Overfitting & Underfitting

Generalisation means predicting well on examples that did not influence model fitting or selection. Training error measures how closely a model matches the observations used to estimate its parameters. Validation error measures performance on separate observations used to compare modelling choices. These errors answer different questions: fitting the available sample is necessary, but it does not guarantee learning a useful prediction rule.

Underfitting occurs when the fitted model fails to capture useful structure. A constant predictor cannot represent a relationship that changes with its input. More generally, unsuitable features, excessive regularisation, or incomplete optimisation can leave even a powerful algorithm underfitted. Training and validation errors are often both high, but “high” must be judged against a suitable baseline, the task requirements, and unavoidable uncertainty in the target.

Overfitting occurs when fitting sample-specific details harms generalisation. Those details may include measurement noise, unusual observations, or accidental associations. Literal memorisation is only one mechanism. A flexible polynomial can bend towards noisy targets without storing a lookup table. Training error may decrease while validation error increases, because the extra flexibility improves agreement with the observed sample rather than the underlying predictive relationship.

Model capacity describes the range of relationships a model can represent. Increasing polynomial degree or allowing deeper decision trees generally increases capacity. Regularisation restricts effective flexibility: for example, ridge regression penalises large coefficients while fitting the training targets. Stronger restrictions can reduce sensitivity to individual observations, but too much restriction can erase useful structure. This is a trade-off, not a rule that simpler models always win.

The bias–variance perspective describes systematic error from restrictive assumptions versus sensitivity to the particular training sample. Neither component can normally be read directly from one pair of scores. A large training–validation gap suggests overfitting, but distribution shift or an unrepresentative split can also produce it. Conversely, similar errors do not prove a model is good. The scikit-learn guide separates metrics, cross-validation, validation curves, learning curves, and leakage prevention because reliable diagnosis requires both an appropriate comparison and a trustworthy evaluation procedure. No single accuracy threshold establishes either condition across different datasets and tasks.

Practical walkthrough

First, define the prediction task and choose a metric. For regression, mean squared error (MSE) averages squared differences between predictions and targets; lower is better. Squaring makes large errors count disproportionately, so check whether that matches your practical costs. Establish a baseline, such as predicting the training-target mean, before interpreting a score as poor.

Second, separate model development from final evaluation. Use training data to fit parameters, validation data to choose settings, and an untouched test set for the final assessment. Split according to how predictions will be used: future observations require time-aware evaluation, while repeated measurements from one person may require grouped splits. Random splitting is not universally appropriate.

Third, compare a small, deliberate range of capacities. Record training and validation error using the same metric. If both remain poor, investigate features, optimisation, and excessive restrictions. If training error improves while validation error worsens, try reduced capacity or stronger regularisation. More representative training data may help an unstable model, but cannot automatically repair a missing predictive feature.

Fourth, use cross-validation when one development split is too fragile. Fit each candidate separately on each training fold and evaluate its held-out fold. Fit scaling, imputation, and feature selection only within that training fold; a scikit-learn pipeline helps enforce this boundary. Cross-validation estimates performance; it does not itself change the learning algorithm.

Finally, inspect variation across folds, not just the average. Choose the configuration, refit on the available development data, and evaluate once on the reserved test set. Report limitations alongside the score rather than repeatedly tuning against that test result.

Worked example

# Python standard library only. Fixed illustrative data.
# Intended signal: y = x; the middle training target has an error.
x_train = [-1.0, 0.0, 1.0]
y_train = [-1.0, 1.0, 1.0]
x_valid = [-0.5, 0.5]
y_valid = [-0.5, 0.5]

# Fit a constant and an ordinary least-squares line on training only.
x_mean = sum(x_train) / len(x_train)
y_mean = sum(y_train) / len(y_train)
slope = sum((x - x_mean) * (y - y_mean)
            for x, y in zip(x_train, y_train)) / sum(
                (x - x_mean) ** 2 for x in x_train)
intercept = y_mean - slope * x_mean

def constant(x):
    return y_mean

def line(x):
    return intercept + slope * x

def quadratic(x):
    # Lagrange interpolation: the quadratic through all training points.
    total = 0.0
    for i in range(len(x_train)):
        term = y_train[i]
        for j in range(len(x_train)):
            if i != j:
                term *= (x - x_train[j]) / (x_train[i] - x_train[j])
        total += term
    return total

def mse(predict, xs, ys):
    return sum((predict(x) - y) ** 2
               for x, y in zip(xs, ys)) / len(xs)

print('model      train_MSE valid_MSE')
for name, predict in [('constant', constant), ('line', line),
                      ('quadratic', quadratic)]:
    train = mse(predict, x_train, y_train)
    valid = mse(predict, x_valid, y_valid)
    print(f'{name:<10} {train:.3f}     {valid:.3f}')

Expected output

model      train_MSE valid_MSE
constant   0.889     0.361
line       0.222     0.111
quadratic  0.000     0.562

The constant ignores changes in x and underfits the intended relationship. Least squares learns slope 1 and intercept 1/3: the corrupted middle target shifts the line upwards. The quadratic learns −x² + x + 1, matching every training target exactly. At both validation inputs, its prediction exceeds the target by 0.75, giving MSE 0.5625 before rounding. Thus perfect training fit loses to an imperfect line. These tiny, deliberately constructed data illustrate mechanics, not a dependable performance estimate. Validation targets are noiseless by design; real targets may contain noise too. No final test score or causal conclusion is claimed.

Common mistakes and how to avoid them

• Choosing the model with the lowest training error.

  Compare held-out validation errors instead. Training error rewards fitting sample-specific details; use the test set only after choosing the model.

• Treating any training–validation gap as proof of overfitting.

  Check for time shifts, different populations, small validation samples, and inconsistent preprocessing. Compare several suitable splits before assigning a diagnosis.

• Fitting preprocessing before splitting the data.

  Learn means, imputation values, and selected features from training observations only. Apply those learned transformations unchanged to validation and test observations.

• Increasing regularisation whenever performance is poor.

  Inspect training error first. If the model already underfits, stronger restrictions may worsen both errors; investigate features, optimisation, or greater capacity instead.

Review questions

Further reading

• scikit-learn user guide