OptimizationConcept reference

Gradient noise

For IID gradient components at a fixed parameter, batch averaging divides variance by batch size and SD by its square root.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
At a fixed parameter, different sampled examples can produce different gradient values
To reduce standard deviation by a factor of k, increase batch size by k²
Variance and standard deviation are different quantities
A closer look
Key takeaway
Sources & further reading
Concept connections

01At a fixed parameter, different sampled examples can produce different gradient values#

At a fixed parameter, different sampled examples can produce different gradient values. Gradient noise describes that variability. Averaging independent samples makes the estimate of the mean direction less variable; it does not alter the variability of each original example.

Fix the parameter and one gradient coordinate. IID examples give IID draws for that coordinate. Averaging b draws preserves the mean and divides variance by b.

\begin{gathered}\operatorname{Var}(\bar g)=\sigma^2/b\\\operatorname{SD}(\bar g)=\sigma/\sqrt b\end{gathered}

Single-example variance 36 and batch size 9 give average variance 4, hence standard deviation 2.

Figure 1At a fixed parameter, IID gradient components have variance 36 and SD 6. Averaging nine gives variance 4 and SD 2. Both spans refer to their common expected gradient; no normal law or coverage probability is asserted.

If one gradient coordinate has variance 36, its standard deviation is 6. Averaging 9 IID draws gives variance $36/9=4$ and standard deviation $\sqrt4=2$ . Variance falls by a factor of 9, while standard deviation falls by a factor of 3. Keep track of which quantity the question asks for.

Check your reasoning

At a fixed parameter, IID draws of one gradient coordinate have variance 12. Find the variance of their batch-3 average.

Show answer and explanation

12/3=4.

02To reduce standard deviation by a factor of k, increase batch size by k²#

To reduce standard deviation by a factor of k, increase batch size by k². Halving SD takes four times the batch size under this model.

For a target SD, square the ratio of individual SD to target SD.

Check your reasoning

At a fixed parameter, IID draws of one gradient coordinate have SD 8. What batch size gives average SD 2?

Show answer and explanation

Square the ratio: (8/2)²=16.

03Variance and standard deviation are different quantities#

Variance and standard deviation are different quantities. Compute average variance first, then take its square root for SD.

This calculation assumes independent draws at a fixed parameter. It does not claim that each raw gradient becomes less variable, or that training quality is guaranteed.

Check your reasoning

At a fixed parameter, IID draws of one gradient coordinate have variance 81. A batch-9 report calls 9 its average SD. Correct the SD.

Show answer and explanation

Average variance 9; SD 3.

04A closer look#

To halve the standard deviation again, this model needs four times as many independent draws. That costs more work and does not by itself ensure better final predictions. Correlated samples do not obey the same simple variance-division rule.

Key takeaway

State the sampling model, divide variance by batch size, and take a square root only when asked for SD.

Interpret batch-size effects under a stated independent-gradient variance model.

Sources & further reading

[1]
D2L §12.5.2 statistical properties of minibatches ↗D2L §12.5.2 statistical properties of minibatches · Article

01At a fixed parameter, different sampled examples can produce different gradient values#

02To reduce standard deviation by a factor of k, increase batch size by k²#

03Variance and standard deviation are different quantities#

04A closer look#

State the sampling model, divide variance by batch size, and take a square root only when asked for SD.

Sources & further reading

Reference this concept