What backpropagation does before gradient descent takes a step

A typical PyTorch training loop calls loss.backward() and then optimizer.step(). Both happen after the model makes a prediction, which makes it easy to treat them as one operation. But you can stop between those calls and inspect something useful: the gradients have been calculated, while the parameters still have their old values.

Backpropagation computes how the loss changes with each parameter. Gradient descent uses those derivatives to choose new parameter values. In PyTorch, the backward call handles the first task; the optimizer handles the second. Its update depends on which optimizer you chose. The PyTorch optimization tutorial makes this separation explicit.

We can follow both tasks with a model small enough to calculate by hand. It has one weight and one bias, so we can account for every number without hiding the distinction inside a library.

Start with a prediction and a loss

Our model predicts $\hat y = wx + b$. Give it the input $x = 2$ with target $y = 1$, and start with weight $w = 1$ and bias $b = 0$. The prediction is

$$\hat y = 1 \times 2 + 0 = 2.$$

The model predicts one unit too high. To measure that error, use the loss $L = \tfrac12(\hat y-y)^2$. The factor of one half makes the derivative simpler; it does not change where this loss is minimized. For the current prediction,

$$L = \tfrac12(2-1)^2 = 0.5.$$

That completes the forward pass. We know what the model predicts and how much loss that prediction incurs. We have not yet calculated how either parameter affects the loss.

Work backward to find the gradients

First consider the last operation, which turns the prediction error into a loss. Its derivative with respect to the prediction is

$$\frac{\partial L}{\partial \hat y} = \hat y-y = 1.$$

At this prediction, a small increase in $\hat y$ would increase the loss at a rate of one unit of loss per unit of prediction. To find how the weight affects the loss, we also need the rate at which the weight changes the prediction. Since $\hat y = wx+b$, that rate is $x = 2$.

The chain rule multiplies these two local rates:

$$\frac{\partial L}{\partial w} = \frac{\partial L}{\partial \hat y} \frac{\partial \hat y}{\partial w} = 1 \times 2 = 2.$$

The bias enters the prediction directly, so its corresponding derivative is one:

$$\frac{\partial L}{\partial b} = \frac{\partial L}{\partial \hat y} \frac{\partial \hat y}{\partial b} = 1 \times 1 = 1.$$

We now have the gradient $(2, 1)$ with respect to $(w, b)$. Both entries are positive, so increasing either parameter slightly would increase the loss from this starting point. The weight has twice the local effect because the input multiplying it is two.

This backward calculation is the small version of what backpropagation does in a larger network. It starts at the loss and applies the chain rule through the operations that produced it, reusing intermediate results. When a value affects the loss through multiple paths, their derivative contributions add. Dive into Deep Learning develops that process for a multilayer model.

The parameters are still $w = 1$ and $b = 0$. Calculating their derivatives has not changed them.

Use the gradients to update the parameters

Plain gradient descent subtracts a scaled gradient from the current parameters. The scale is the learning rate, written $\eta$. With $\eta = 0.1$, the updates are

$$\begin{aligned} w_{\text{new}} &= 1 - 0.1 \times 2 = 0.8,\\ b_{\text{new}} &= 0 - 0.1 \times 1 = -0.1. \end{aligned}$$

Both derivatives came from the same starting state. We do not update the weight and then recalculate the bias derivative halfway through this step.

Using the new parameters, the model predicts $0.8 \times 2 - 0.1 = 1.5$. Its new loss is $\tfrac12(1.5-1)^2 = 0.125$, down from $0.5$. This particular update improved the prediction on our single example.

The second row is where the distinction becomes visible. Backpropagation gives us additional information about the current parameters; gradient descent then changes those parameters.

Changing the learning rate does not change this starting gradient

Suppose we return to the original parameters and choose $\eta = 0.5$. The derivatives at that point remain $(2, 1)$, but the update now gives $w = 0$ and $b = -0.5$. The prediction becomes $-0.5$, and the loss rises to $1.125$. We moved in the direction suggested by the gradient and still made the loss worse because we went too far.

For this example, the prediction after an update is $2-5\eta$, so the resulting loss is $\tfrac12(1-5\eta)^2$. The curve below shows how much the outcome depends on the step size.

Figure link ↗SVG ↓PNG ↓Data ↓

A learning rate of $0.2$ happens to fit this one example exactly. That is a property of these numbers, not a rate to copy into a neural network. With a larger model and more data, the useful step size depends on the loss surface and the optimization method. Our explanation of gradient descent oscillation examines another case where the step size determines whether repeated updates converge.

Reproduce the whole step in Python

This code calculates the derivatives explicitly, so it needs no machine learning library. The backward section uses the chain rule we worked out above; an automatic differentiation system performs that bookkeeping for a larger graph.

x, target = 2.0, 1.0
w, b = 1.0, 0.0
eta = 0.1

# Forward pass at the current parameters.
prediction = w * x + b
residual = prediction - target
loss = 0.5 * residual ** 2

# Backward pass: calculate both derivatives before updating.
grad_w = residual * x
grad_b = residual

# Plain gradient descent.
w, b = w - eta * grad_w, b - eta * grad_b
new_prediction = w * x + b
new_loss = 0.5 * (new_prediction - target) ** 2

print(f"gradient: ({grad_w:.1f}, {grad_b:.1f})")
print(f"parameters: ({w:.1f}, {b:.1f})")
print(f"loss: {loss:.3f} -> {new_loss:.3f}")

It prints a gradient of (2.0, 1.0), parameters of (0.8, -0.1), and a loss change from 0.500 to 0.125.

In a PyTorch implementation, loss.backward() accumulates gradients in the parameters' .grad attributes. A usual training loop clears old gradients before calculating a fresh step, unless it deliberately accumulates them across batches. The autograd tutorial explains that behavior. Clearing a gradient is another separate operation; it does not reset the learned weight.

What changes when you use a different optimizer?

The gradient calculation can stay the same when the update rule changes. An optimizer such as SGD with momentum or Adam uses gradients together with its own state to determine the update. It therefore does more than subtract the current gradient multiplied by a single learning rate. These optimizers occupy the same part of the training loop, after the backward pass.

This separation is useful when debugging. If the derivatives are wrong, inspect the loss calculation and the graph used to differentiate it. If the derivatives are right but the loss grows after an update, examine how the optimizer turned them into parameter changes. In our example, changing only the learning rate was enough to turn an improving step into a harmful one.