Gradient descent steps

01Keep the state, gradient, and step separate#

Gradient descent uses a local derivative to choose a new parameter state. Let $\theta$ denote the current parameter vector, $f$ the scalar objective, and $g=\nabla f(\theta)$ its gradient at that state.

The learning rate $\eta$ is a positive number controlling the scale of the update. One plain gradient descent step is

There are three different objects here: the gradient $g$, the change $-\eta g$, and the new parameter $\theta_{\mathrm{new}}$. A gradient is a vector of rates; it is not itself the new parameter value.

The minus sign chooses the direction of local decrease. The learning rate decides how far we move in that direction.

02Update each coordinate#

Take $\theta=(4,-1)^T$, gradient $g=(2,-4)^T$, and learning rate $\eta=0.5$. First scale the gradient: $\eta g=(1,-2)^T$. Then subtract:

The first coordinate decreases. The second increases because we subtract a negative gradient component.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

The actual step is $\theta_{\mathrm{new}}-\theta=(-1,2)^T$, the opposite direction to the gradient. On equal coordinate scales, its length is the gradient length multiplied by $0.5$.

Do not subtract the learning rate alone from each coordinate. It scales the corresponding derivative, so coordinates can change by different amounts and in different directions.

03Calculate the gradient from an objective#

Consider the two-parameter objective $f(a,b)=(a-1)^2+2b^2$. Its gradient is

At $(a,b)=(3,-1)$, the objective is $6$ and the gradient is $(4,-4)^T$. With $\eta=0.1$, the update gives $(a_{\mathrm{new}},b_{\mathrm{new}})=(2.6,-0.6)$.

Evaluate the objective at the new state to see what happened:

This particular step reduced the objective from $6$ to $3.28$. Calculating the new value is a separate forward evaluation; the old gradient alone is not the new loss.

The same rule works with one parameter. For $h(x)=2(x-1)^2$, the derivative is $h'(x)=4(x-1)$. At $x=3$, it is $8$, so a learning rate of $0.1$ gives $x_{\mathrm{new}}=3-0.1(8)=2.2$.

The objective also happens to equal $8$ at this starting point. That numerical coincidence does not make objective values interchangeable with derivatives.

04Use derivatives from one starting state#

A plain gradient step computes all gradient entries at the same current parameter state, then updates all coordinates. For our two-parameter example, both derivatives came from $(3,-1)$.

Updating one coordinate and recomputing another derivative midway would be a different procedure for objectives whose variables interact.

After reaching $(2.6,-0.6)$, the next gradient is $(3.2,-2.4)^T$. It differs from the original $(4,-4)^T$ because the parameters have moved.

The next iteration therefore starts with a new derivative calculation at the new state. Keep the subscript or evaluation point visible when following more than one step.

05Understand what the direction does and does not guarantee#

Return to the original $(3,-1)$ and the same gradient $(4,-4)^T$, but choose $\eta=1$. The new point is $(-1,3)$.

Its objective is $(-2)^2+2(3^2)=22$, much larger than the starting value $6$. The local downhill direction was correct; the step was too large.

For a differentiable objective with a nonzero gradient, a sufficiently small positive step along the negative gradient decreases the objective locally. This does not guarantee that an arbitrary learning rate decreases it, or that repeated steps find a global minimum.

A zero gradient produces no movement under this rule. A stationary point may be a minimum, maximum, or saddle; the update formula alone does not classify it.

For a weight matrix, the same operation happens entry by entry:

The gradient has the same shape as $W$, so every parameter receives its own scaled derivative. Backpropagation computes those derivatives. Plain gradient descent uses them to change the parameters; momentum and Adam use other update rules.