Adam updates

Adam scales each corrected first moment by the square root of its corrected second moment plus the stated ε.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
Understand the idea
For a vector, use each coordinate’s own corrected moments
The numerator is the corrected first moment, which includes history
For a neural-network weight vector, each coordinate has its own two moments
Key takeaway
Sources & further reading
Concept connections

01Understand the idea#

Adam is an optimizer that combines gradient memory with a separate update scale for each parameter. Its corrected first moment is a signed average of gradients; its corrected second moment summarizes squared gradient size. Together they determine a proposed update.

The hats on $\hat m$ and $\hat v$ mean the initialization correction has already been applied. In this lesson those corrected values are supplied. You will combine them, the learning rate, and epsilon to find the new parameter, without rebuilding the earlier moment history.

Adam combines a corrected first moment with coordinatewise scaling from a corrected second moment. Here both corrected moments are already supplied:

\theta_j\prime=\theta_j-\eta\frac{\hat m_j}{\sqrt{\hat v_j}+\epsilon}

With θ=5, m̂=4, v̂=9, η=0.2 and ε=1, the divisor is 4. The step is 0.2, so θ′=4.8.

Figure 1At θ=5, corrected moments m=4,v=9, rate .2 and epsilon 1 give denominator sqrt(9)+1=4. The displacement is −.2×4/4=−.2 and the new parameter is 4.8. Horizontal distances are numerical.

For $\theta=5$ , $\hat m=4$ , $\hat v=9$ , $\eta=0.2$ , and $\epsilon=1$ , compute the denominator first: $\sqrt9+1=4$ . The normalized moment is $4/4=1$ , the step is $0.2(1)=0.2$ , and the new parameter is $5-0.2=4.8$ . Epsilon is deliberately large here to make its location visible.

Check your reasoning

Use $\theta\prime=\theta-\eta\hat m/(\sqrt{\hat v}+\epsilon)$ . Corrected moments supplied: (θ,m̂,v̂,η,ε)=(3, 8, 9, 0.5, 1). Find θ′.

Show answer and explanation

Divisor 4; new parameter 2.

02For a vector, use each coordinate’s own corrected moments#

For a vector, use each coordinate’s own corrected moments. The first moment supplies a signed direction; the square-root second moment supplies scale.

These inputs have already been bias corrected. Do not divide them by a correction factor a second time.

Check your reasoning

Use $\theta\prime=\theta-\eta\hat m/(\sqrt{\hat v}+\epsilon)$ . Corrected: θ= $(5, 2)$ , m̂= $(6, -4)$ , v̂= $(4, 1)$ , η=0.5, ε=1. Find θ′.

A $(4, 3)$
B $(4, 4)$
C $(5, 3)$

Show answer and explanation

$(4, 3)$

Steps (1,−1); θ′=(4,3).

03The numerator is the corrected first moment, which includes history#

The numerator is the corrected first moment, which includes history. Substituting the current gradient would produce a different update.

This lesson computes the stated step. The formula alone does not establish that Adam outperforms every other optimizer or guarantees convergence in every setting.

Check your reasoning

Use $\theta\prime=\theta-\eta\hat m/(\sqrt{\hat v}+\epsilon)$ . Corrected (θ,m̂,v̂,η,ε)=(4, 2, 1, 0.5, 1); current g=6. Draft uses g as numerator. Correct θ′.

Show answer and explanation

3.5

Use m̂=2, not g=6: θ′=3.5.

04For a neural-network weight vector, each coordinate has its own two moments#

For a neural-network weight vector, each coordinate has its own two moments. A negative first moment can increase the corresponding weight because the update subtracts a negative number. Adam still has a learning rate and is not guaranteed to outperform every other optimizer on every task.

Key takeaway

Use the supplied corrected moments once, preserve ε outside the root, and update each coordinate.

Compute an Adam parameter update from supplied bias-corrected moment estimates.

Sources & further reading

[1]
Kingma & Ba Algorithm1 parameter update ↗Kingma & Ba Algorithm1 parameter update · Article

01Understand the idea#

02For a vector, use each coordinate’s own corrected moments#

03The numerator is the corrected first moment, which includes history#

04For a neural-network weight vector, each coordinate has its own two moments#

Use the supplied corrected moments once, preserve ε outside the root, and update each coordinate.

Sources & further reading

Reference this concept