Skip to content
Glacius
OptimizationConcept reference

Gradient descent steps

One gradient descent update subtracts the learning rate times the current gradient.

2 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. Gradient descent updates parameters by moving against the loss’s local sensitivity
  3. A closer look
  4. Adding a nonzero gradient gives an ascent direction
  5. Training repeats this operation using a gradient evaluated at the current parameter
  6. Key takeaway
  7. Sources & further reading
  8. Concept connections

01Gradient descent updates parameters by moving against the loss’s local sensitivity#

Gradient descent updates parameters by moving against the loss’s local sensitivity. A positive derivative means increasing that parameter raises the loss locally, so the update subtracts it. The learning rate controls how much of this direction to take.

Gradient descent moves opposite the current gradient when minimizing an objective. Scale the gradient by a positive learning rate, then subtract:

θnew=θ−η∇f(θ)\theta_{\mathrm{new}}=\theta-\eta\nabla f(\theta)

At θ=(4,−1), gradient (2,−4) and rate 0.5 give a scaled gradient (1,−2). Subtract it to get (3,1).

Current parameter θ=(4,−1). Gradient g=(2,−4) and rate η=.5 give step −ηg=(−1,2). The step arrow starts at θ and ends at the new parameter (3,1). Both coordinate axes use equal scale.Current parameter θ=(4,−1). Gradient g=(2,−4) and rate η=.5 give step −ηg=(−1,2). The step arrow starts at θ and ends at the new parameter (3,1). Both coordinate axes use equal scale.
Figure 1Current parameter θ=(4,−1). Gradient g=(2,−4) and rate η=.5 give step −ηg=(−1,2). The step arrow starts at θ and ends at the new parameter (3,1). Both coordinate axes use equal scale.
Link to this figure ↗Download SVGDownload PNG

With parameter (3,−2)(3,-2), gradient (4,−6)(4,-6), and learning rate 0.1, first scale the gradient to (0.4,−0.6)(0.4,-0.6). Then subtract coordinatewise: (3−0.4,−2−(−0.6))=(2.6,−1.4)(3-0.4,-2-(-0.6))=(2.6,-1.4). Subtracting a negative gradient increases that parameter. The gradient itself is not the new parameter.

Check your reasoning

Current θ=(3,2)\theta=(3, 2), gradient g=(4,−8)g=(4, -8), η=0.25\eta=0.25. Find the descent update.

  1. A(2,5)(2, 5)
  2. B(2,4)(2, 4)
  3. C(3,4)(3, 4)
Show answer and explanation
(2,4)(2, 4)

Subtract ηg\eta g coordinatewise: (2,4)(2, 4).

02A closer look#

If the gradient is not supplied, differentiate the stated objective and evaluate it at the current parameter.

Use that current gradient in the update. Do not substitute the objective value in place of its derivative.

Check your reasoning

f(x)=4(x−3)2/2f(x)=4(x-3)^2/2. Start x=5x=5, rate η=0.25\eta=0.25. Find the next x.

Show answer and explanation
3

Derivative 4(x−3) gives 8; new x is 5−0.25(8)=3.

03Adding a nonzero gradient gives an ascent direction#

Adding a nonzero gradient gives an ascent direction. Subtracting an unscaled gradient silently uses a learning rate of 1.

A correctly computed update is one step. Its effect on loss still depends on the step size and objective.

Check your reasoning

Current θ=(2,−3)\theta=(2, -3), g=(6,−2)g=(6, -2), η=0.5\eta=0.5. Draft update: (5,−4)(5, -4). Repair it.

  1. A(−1,−1)(-1, -1)
  2. B(0,−2)(0, -2)
  3. C(−1,−2)(-1, -2)
Show answer and explanation
(−1,−2)(-1, -2)

Subtract ηg\eta g coordinatewise: (−1,−2)(-1, -2).

04Training repeats this operation using a gradient evaluated at the current parameter#

Training repeats this operation using a gradient evaluated at the current parameter. After moving, the old gradient is generally no longer the current one. The method needs a sensible step size; the minus sign alone does not guarantee the next loss is smaller.

Key takeaway

Evaluate the current gradient, scale every coordinate by the rate and subtract from the current parameter.

  • Compute one gradient descent update.

Sources & further reading

  1. [1]
    Dive into Deep Learning §12.3.1–12.3.2 ↗Dive into Deep Learning §12.3.1–12.3.2 · Article

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Gradient descent steps.” Math behind ML. /learn/o-gradient-step