Gradient Descent

Understanding Gradient Descent

Gradient descent adjusts a model’s parameters by repeatedly asking: which small change would reduce the current training loss? The model specifies how inputs become predictions; the loss measures prediction error; the optimizer changes parameters. These are separate choices. For example, linear regression can use squared error with gradient descent, but it can also be solved using other numerical methods. Gradient descent is not itself a prediction model.

For one parameter w, the derivative dL/dw measures how loss changes locally as w increases. A positive derivative means increasing w raises loss locally, so the update should decrease w. A negative derivative calls for increasing w. With several parameters, the gradient collects their partial derivatives into a vector. The update is θ_new = θ_old − η∇L(θ_old), where η is a positive learning rate. All gradient components are evaluated at the same current parameter values.

The gradient describes local behavior, not the whole loss surface. Consequently, moving against it reduces a smooth loss only when the step is suitably small. The learning rate multiplies the gradient: it is not the distance traveled independently of gradient magnitude. Large gradients produce larger updates at the same learning rate. An oversized step can cross a minimum and increase loss; an extremely small step can make progress unnecessarily slow.

Batch gradient descent computes each update using the entire training set. Stochastic gradient descent uses one training example per update; mini-batch gradient descent uses a subset. Smaller batches make updates cheaper but introduce variation into gradient estimates, so individual updates need not reduce the full training loss. An epoch means one pass through the training examples, not necessarily one update. The supplied scikit-learn guide treats stochastic gradient descent as supporting both classification and regression.

For convex squared-error linear regression, suitable steps can approach a global minimum. Neural-network losses are generally nonconvex, so equivalent guarantees do not apply. A small gradient can indicate a minimum, a flat region, or a saddle point. Moreover, minimizing training loss does not establish good predictions on unseen data. Adam adapts updates using gradient history, but it is not universally better than SGD; optimizer choice still requires validation.

Practical walkthrough

Use Python 3.8 or later; no packages are required. Familiarity with loops, functions, and squared errors is sufficient. The example deliberately uses two training observations, (−1, −2) and (1, 2), and the model prediction = w × x. There is no intercept: we assume predictions should pass through the origin. This restriction makes every calculation inspectable, but would be inappropriate for data requiring an offset.

First, define mean squared error as L(w) = mean((w × x − y)²). Differentiate with respect to w to obtain dL/dw = mean(2 × x × (w × x − y)). The factor x comes from differentiating the prediction; the factor 2 comes from the square. Both observations contribute before the parameter changes, making this batch gradient descent.

Next, initialize w to zero and choose learning rate 0.25. For these observations, the loss simplifies to (w − 2)² and the gradient to 2(w − 2). The first gradient is −4, so subtracting 0.25 times that gradient moves w to 1. Each subsequent update halves the remaining distance to 2.

Print the initial state and then four updated states. A fixed iteration count makes this demonstration reproducible; practical training also needs stopping criteria, such as negligible improvement over several checks and a maximum iteration budget.

Treat the printed losses only as training diagnostics. No test performance is estimated here. For real experiments, split data before fitting preprocessing, learn scaling from training data only, and apply that same transformation to validation and test data. Select learning rates using validation results, leaving the test set untouched until final evaluation.

Worked example

train = ((-1.0, -2.0), (1.0, 2.0))
learning_rate = 0.25
w = 0.0


def loss(weight):
    return sum((weight * x - y) ** 2 for x, y in train) / len(train)


def gradient(weight):
    return sum(2 * x * (weight * x - y) for x, y in train) / len(train)


print("step w loss")
print(f"0 {w:.6f} {loss(w):.6f}")
for step in range(1, 5):
    grad = gradient(w)
    w = w - learning_rate * grad
    print(f"{step} {w:.6f} {loss(w):.6f}")

Expected output

step w loss
0 0.000000 4.000000
1 1.000000 1.000000
2 1.500000 0.250000
3 1.750000 0.062500
4 1.875000 0.015625

The initial predictions are both zero, producing squared errors of four. After one update, w equals one, and each squared error equals one. Because every update halves the parameter error w − 2, it quarters the loss. The last printed weight, 1.875, is closer to the optimum but has not reached it. Six decimal places control display formatting rather than optimization precision. All values in this short run are exactly representable in ordinary binary floating-point arithmetic, so the output is deterministic. The symmetric observations simplify the algebra; they do not demonstrate robustness to noise or performance on a realistic prediction task.

Common mistakes and how to avoid them

• Adding the gradient when trying to minimize loss.

  Subtract learning_rate × gradient. Check the sign manually: here the initial negative gradient must increase w toward two.

• Using a gradient inconsistent with the reported loss.

  Differentiate the exact objective, including its averaging convention. Replacing a mean with a sum changes gradient magnitude and therefore the effective step size.

• Increasing iterations when the loss is exploding.

  Check the derivative and reduce the learning rate first. Inspect feature scales; if standardizing, fit the scaler only on training observations.

• Equating lower training loss with a better model.

  Monitor separate validation results for overfitting. Compare optimizers under the same evaluation protocol rather than assuming an adaptive method must win.

Review questions

Further reading

• scikit-learn user guide