Moment bias correction

Bias correction divides a zero-initialized EMA by the total weight accumulated through its current step.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
Understand the idea
If gradients are supplied instead of the raw moment, compute the recurrence first
With a constant gradient, correction recovers that constant exactly
Adam applies this correction separately to its first and second moving moments
Key takeaway
Sources & further reading
Concept connections

01Understand the idea#

An exponential moving average started at zero initially gives its observations less than total weight 1. Even a constant nonzero signal is pulled toward that artificial starting zero. Bias correction divides by the accumulated observation weight to remove this initialization effect.

A zero-initialized exponential moving average starts with missing weight. With decay β and step t≥1, its accumulated weight is 1−βᵗ.

\hat m_t=\frac{m_t}{1-\beta^t}

Assume 0≤β<1 and a zero start.

With β=0.5 at t=2, accumulated weight is 0.75. A raw moment of 9 becomes a corrected moment of 12.

Figure 1Starting an EMA at zero with beta .5 leaves observation weights .25 and .5 at t=2, total .75. Dividing by .75 renormalizes them to 1/3 and 2/3. The lesson’s raw moment 9 therefore becomes 12 after bias correction.

With decay 0.5 and constant gradients 12, the first raw moment is 6. The second is $0.5(6)+0.5(12)=9$ . At step 2, accumulated weight is $1-0.5^2=0.75$ . Dividing $9/0.75$ recovers 12. Dividing by $1-0.5$ instead would use the wrong update count.

Check your reasoning

Zero-initialized EMA: m_t=5.25, β=0.5, t=3. Find m̂=m_t/(1−β^t).

Show answer and explanation

Divide by 0.875: 6.

02If gradients are supplied instead of the raw moment, compute the recurrence first#

If gradients are supplied instead of the raw moment, compute the recurrence first:

\begin{gathered}m_t=\beta m_{t-1}\\{}+(1-\beta)g_t\end{gathered}

Divide the final raw moment by its accumulated weight, using the actual update count.

Check your reasoning

Start m0=0. Use m_t=βm_(t−1)+(1−β)g_t with β=0.5, (g1,g2)=(3, 6). Find m̂2=m2/(1−β²).

Show answer and explanation

m1=1.5, m2=3.75; corrected=5.

03With a constant gradient, correction recovers that constant exactly#

With a constant gradient, correction recovers that constant exactly. The smaller raw value reflects missing initial weight.

For changing gradients, correction normalizes their accumulated weights. It does not promise to equal the current gradient or remove bias from a drifting target.

Check your reasoning

Zero-start EMA, β=0.5; gradients all 8 for 2 steps. Draft reports raw m=6 as corrected. Give m̂=m/(1−β^2).

Show answer and explanation

Raw 6 has weight 0.75. Correction gives 8.

04Adam applies this correction separately to its first and second moving moments#

Adam applies this correction separately to its first and second moving moments. A corrected average still combines past observations; with a changing gradient, it need not equal the current gradient. Correction removes the zero-start weighting effect, not every source of estimation error.

Key takeaway

Compute the raw EMA if needed, then divide by 1−βᵗ using the actual step count.

Correct a zero-initialized exponential moving average using a supplied formula.

Sources & further reading

[1]
Kingma & Ba Algorithm1 and §3 ↗Kingma & Ba Algorithm1 and §3 · Article

01Understand the idea#

02If gradients are supplied instead of the raw moment, compute the recurrence first#

03With a constant gradient, correction recovers that constant exactly#

04Adam applies this correction separately to its first and second moving moments#

Compute the raw EMA if needed, then divide by 1−βᵗ using the actual step count.

Sources & further reading

Reference this concept