Adaptive scaling

A supplied second moment rescales its coordinate gradient through the stated square-root divisor.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
Different model parameters can experience very different gradient scales
Step size depends on both gradient magnitude and its coordinate divisor
A second moment is not a standard deviation
The second moment tracks squared gradient size, not centered variance
Key takeaway
Sources & further reading
Concept connections

01Different model parameters can experience very different gradient scales#

Different model parameters can experience very different gradient scales. Adaptive methods use a separate running scale estimate for each coordinate, so one large-gradient coordinate need not dominate every update. The denominator in these tasks is supplied, not estimated from the current gradient alone.

Adaptive scaling gives each coordinate its own divisor. Here the supplied second moments satisfy s≥0, ε sits outside the root, and every denominator must be positive:

\theta_j\prime=\theta_j-\eta\frac{g_j}{\sqrt{s_j}+\epsilon}

For θ=(4,3), g=(2,4), s=(1,9), η=1 and ε=1, the divisors are (2,4). Equal steps (1,1) give θ′=(3,2).

Figure 1θ=(4,3), gradient (2,4), second moments (1,9), rate 1 and epsilon 1 give denominators (2,4). Both normalized gradient components are 1. Subtracting them moves θ to (3,2), despite unequal raw gradient components.

For parameter $(4,3)$ , gradient $(2,4)$ , second moments $(1,9)$ , rate 1, and $\epsilon=1$ , take square roots $(1,3)$ and add epsilon to get divisors $(2,4)$ . Divide matching gradient entries to get $(1,1)$ . Subtract from the parameter to obtain $(3,2)$ .

Check your reasoning

Use $\theta\prime=\theta-\eta g/(\sqrt{s}+\epsilon)$ coordinatewise. θ= $(3, 5)$ , g= $(4, 6)$ , s= $(1, 4)$ , η=0.5, ε=1. Find θ′.

A $(2, 5)$
B $(2, 4)$
C $(3, 4)$

Show answer and explanation

$(2, 4)$

Steps: (1,1).

02Step size depends on both gradient magnitude and its coordinate divisor#

Step size depends on both gradient magnitude and its coordinate divisor. Equal gradients can produce unequal steps when their supplied moments differ.

Compare absolute step magnitudes when direction is irrelevant; a negative gradient still creates a positive magnitude.

Check your reasoning

Use $\theta\prime=\theta-\eta g/(\sqrt{s}+\epsilon)$ coordinatewise. g= $(3, -3)$ , s= $(4, 16)$ , η=1, ε=0. Find |step1|/|step2|.

Show answer and explanation

Magnitudes 1.5 and 0.75; ratio 2.

03A second moment is not a standard deviation#

A second moment is not a standard deviation. Apply the square root in the stated rule, then add ε in the stated location.

Some algorithms place ε differently. Do not silently substitute another convention or treat an uncentered second moment as centered variance.

Check your reasoning

Use $\theta\prime=\theta-\eta g/(\sqrt{s}+\epsilon)$ coordinatewise. (θ,g,s,η,ε)=(5, 8, 9, 1, 1). Draft uses s+ε without √s. Correct θ′.

Show answer and explanation

Root s first, add ε, then divide: θ′=3.

04The second moment tracks squared gradient size, not centered variance#

The second moment tracks squared gradient size, not centered variance. The root returns it to gradient-scale units; epsilon keeps the stated divisor positive. Its placement inside or outside the root changes the arithmetic, so use the exact convention shown.

Key takeaway

Use each coordinate’s own s, preserve the stated ε location, and subtract the resulting scaled gradient.

Compute an update with supplied per-coordinate second-moment scaling.

Sources & further reading

[1]
D2L §12.8.1, alongside Adam Algorithm1 for epsilon outside the square root ↗D2L §12.8.1, alongside Adam Algorithm1 for epsilon outside the square root · Article
[2]
Kingma & Ba Algorithm1 ↗Kingma & Ba Algorithm1 · Article

01Different model parameters can experience very different gradient scales#

02Step size depends on both gradient magnitude and its coordinate divisor#

03A second moment is not a standard deviation#

04The second moment tracks squared gradient size, not centered variance#

Use each coordinate’s own s, preserve the stated ε location, and subtract the resulting scaled gradient.

Sources & further reading

Reference this concept