Momentum

Momentum updates a stored velocity and steps opposite that new velocity.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
Momentum gives an optimizer memory of earlier gradients
For vector parameters, update each velocity coordinate first
Stored history can outweigh the current gradient
The gradient now points negatively, yet this example’s memory still points positively
Key takeaway
Sources & further reading
Concept connections

01Momentum gives an optimizer memory of earlier gradients#

Momentum gives an optimizer memory of earlier gradients. Repeatedly aligned gradients can reinforce each other; gradients that alternate signs can partly cancel in that memory. The stored velocity is a running direction, and the parameter update uses its new value.

Momentum retains a velocity from previous gradients. Here we use this explicit convention:

\begin{gathered}v\prime=\beta v+g\\\theta\prime=\theta-\eta v\prime\end{gathered}

This velocity adds g directly; there is no extra (1−β) multiplier.

With old velocity 4, β=0.5 and g=−1, the new velocity is 1. From θ=5 and η=0.2, the parameter becomes 4.8.

Figure 1With v_new=.5v_old+g, old velocity 4 contributes 2 and gradient −1 contributes −1. The resulting velocity is 1. At θ=5 and rate .2, subtracting .2 times that velocity gives 4.8.

Using $v_{new}=\beta v+g$ , let old velocity be 4, $\beta=0.5$ , and current gradient be $-1$ . Retained history is 2; adding the new gradient gives velocity 1. With parameter 5 and rate 0.2, subtract $0.2(1)$ to get 4.8. Using the old velocity would give a different update.

Check your reasoning

Use $v\prime=\beta v+g$ , $\theta\prime=\theta-\eta v\prime$ . Given (θ,v,g,β,η)=(2, 4, 2, 0.5, 0.25). Find (v′,θ′).

A $(4, 2)$
B $(4, 1)$
C $(5, 1)$

Show answer and explanation

$(4, 1)$

New velocity 4; parameter 1.

02For vector parameters, update each velocity coordinate first#

For vector parameters, update each velocity coordinate first. Then subtract the scaled new velocity from its matching parameter coordinate.

The learning rate scales the parameter step, not the stored velocity in this convention.

Check your reasoning

Use $v\prime=\beta v+g$ , $\theta\prime=\theta-\eta v\prime$ . θ= $(3, 4)$ , v= $(2, 4)$ , g= $(1, -1)$ , β=0.5, η=0.5. Find θ′.

A $(2, 3.5)$
B $(2, 4.5)$
C $(3, 3.5)$

Show answer and explanation

$(2, 3.5)$

New velocity $(2, 1)$ ; subtract η times it.

03Stored history can outweigh the current gradient#

Stored history can outweigh the current gradient. Momentum can therefore move with the current gradient on an individual step.

Read the stated recurrence before calculating: other momentum conventions scale their stored state differently.

Check your reasoning

Use $v\prime=\beta v+g$ , $\theta\prime=\theta-\eta v\prime$ . (θ,v,g,β,η)=(4, 8, -1, 0.5, 0.5). Draft: θ must move against g. Compute θ′.

Show answer and explanation

2.5

Velocity 3 carries history; θ′=2.5.

04The gradient now points negatively, yet this example’s memory still points positively#

The gradient now points negatively, yet this example’s memory still points positively. Momentum can therefore disagree with the current gradient on an individual step. Different implementations define velocity differently, so begin with the recurrence supplied by the task.

Key takeaway

Use the stated convention: update velocity first, then subtract the learning-rate-scaled new velocity.

Compute a momentum update using a stated velocity convention.

Sources & further reading

[1]
D2L §12.6.1.3, equation12.6.5 ↗D2L §12.6.1.3, equation12.6.5 · Article

01Momentum gives an optimizer memory of earlier gradients#

02For vector parameters, update each velocity coordinate first#

03Stored history can outweigh the current gradient#

04The gradient now points negatively, yet this example’s memory still points positively#

Use the stated convention: update velocity first, then subtract the learning-rate-scaled new velocity.

Sources & further reading

Reference this concept