Bias vs Variance

Understanding Bias vs Variance

Bias and variance describe two different ways a learning algorithm can miss a prediction target. Imagine repeatedly collecting training datasets from the same process, fitting the same algorithm, and predicting at one fixed input. Bias concerns where those predictions are centered; variance concerns how much they change between datasets. They describe the learning procedure, including its features and settings, rather than just one fitted model.

For regression, let f(x) be the mean target value at input x. Bias at x is the algorithm’s average prediction minus f(x). A straight-line model can have substantial bias when the underlying relationship is curved: changing its coefficients cannot give it the missing curvature. However, simplicity does not automatically imply harmful bias. A straight line is appropriate when the relationship really is linear.

Variance is the spread of predictions around their average across training datasets. A flexible model may follow small fluctuations in observed targets, so replacing a few training observations changes its predictions substantially. For nearest-neighbor regression, averaging only one nearby target preserves local detail but also preserves that target’s noise. Averaging more neighbors smooths noise, while potentially blending together locations with different underlying target values.

Under squared-error loss, expected prediction error at a fixed input equals squared bias plus variance plus irreducible noise variance. This assumes a fresh target with conditional mean f(x), finite variances, and training data independent of the fresh observation. The expectation averages over training datasets and fresh target noise. It is not an identity that splits one observed test score into three directly measurable quantities. Other losses, including classification error, need different treatments.

The trade-off is therefore about expected performance on unseen observations, not minimizing training error. Stronger regularization often reduces variance while increasing bias; more representative training data can reduce variance without forcing a simpler model. Neither pattern is universal. Bagging and random forests often stabilize unstable learners through averaging, especially when their errors are not perfectly correlated. Boosting changes the fitted function sequentially and can address underfitting, but it is not simply a variance-removal device. No ensemble guarantees low bias, eliminates noise, or makes evaluation unnecessary.

Practical walkthrough

Start by defining the prediction task and loss. For numeric targets, mean squared error makes the classical decomposition relevant. Establish a baseline, such as predicting the training-target mean. Decide how future observations will arrive: random splits may suit independent observations, whereas time-ordered or grouped data require splits that respect those dependencies.

Reserve a final test set before comparing model settings. Use cross-validation within the remaining development data to select complexity. Any learned scaling, imputation, or feature selection must be fitted separately on each training fold, then applied unchanged to its validation fold. A scikit-learn pipeline helps enforce this boundary. The supplied guide’s sections on cross-validation, validation curves, learning curves, pipelines, and leakage provide the relevant workflow topics.

Compare training and validation errors across a small range of complexity settings. High errors on both suggest underfitting, weak features, or substantial noise. Low training error alongside much higher validation error suggests overfitting, but distribution differences or leakage-related evaluation mistakes also deserve investigation. These are diagnostic patterns, not direct measurements of bias and variance.

Next, compare performance as training size increases. Improving validation performance with more data supports trying additional representative observations. Persistently poor performance on both partitions supports revisiting features, model capacity, or optimization. Check variation across valid folds rather than trusting one favorable split.

Finally, inspect the controlled example below. It enumerates two equally likely training outcomes, so prediction bias and variance can actually be calculated. After selecting a model in a real project, evaluate it once on the untouched test set and report its limitations.

Worked example

from statistics import mean

# Fixed inputs; underlying mean function is f(x) = x**2.
xs = (-1.0, 0.0, 1.0)
# Only the middle training target has noise: -1 or +1,
# each with probability 1/2. Endpoint targets are exact.
training_targets = ((1.0, -1.0, 1.0), (1.0, 1.0, 1.0))
query = 0.0
truth = query ** 2


def predict(ys, k):
    nearest = sorted(range(len(xs)), key=lambda i: abs(xs[i] - query))[:k]
    return mean(ys[i] for i in nearest)


print('k  mean   bias^2 variance latent_MSE')
for k in (1, 3):
    predictions = [predict(ys, k) for ys in training_targets]
    average = mean(predictions)
    bias_squared = (average - truth) ** 2
    variance = mean((p - average) ** 2 for p in predictions)
    mse = mean((p - truth) ** 2 for p in predictions)
    print(f'{k}  {average:.3f}  {bias_squared:.3f}  {variance:.3f}    {mse:.3f}')

Expected output

k  mean   bias^2 variance latent_MSE
1  0.000  0.000  1.000    1.000
3  0.667  0.444  0.111    0.556

With one neighbor, the prediction is either −1 or +1. Its average equals the true mean, zero, but its variance is one. With three neighbors, predictions are one-third and one: averaging reduces sensitivity to the noisy middle observation but introduces upward bias by including the endpoints. Squared bias plus variance equals the displayed latent-target MSE, apart from rounding. These are exact expectations over the specified two-outcome experiment, not estimated training metrics. No preprocessing or validation-based selection occurs. If predicting a fresh noisy target at zero with independent noise variance one, add one to each MSE. Results describe only this input and design.

Common mistakes and how to avoid them

• Calling the train–validation error gap “the variance.”

  Treat the gap as an overfitting warning. Variance concerns predictions across training datasets; inspect refitted predictions on common inputs and consider split uncertainty.

• Reducing complexity whenever validation error is high.

  Check training error too. If both errors are high, further simplification may worsen underfitting; test better features or greater capacity using validation data.

• Selecting settings repeatedly against the final test set.

  Move selection into cross-validation on development data. Keep the test set untouched, and fit all learned preprocessing only within training folds.

• Assuming every ensemble reduces variance while retaining low bias.

  Distinguish averaging methods from sequential boosting. Evaluate ensemble capacity and regularization; correlated errors limit averaging benefits, and shared systematic errors can remain.

Review questions

Further reading

• scikit-learn user guide