Minibatch gradients

The gradient of a minibatch-average loss is the average of its per-example gradients.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
A minibatch combines several selected examples into one training update
If a batch selects only some rows, average exactly those rows
A gradient sum differs from a gradient average by the batch count
A mean loss uses a mean gradient
Key takeaway
Sources & further reading
Concept connections

01A minibatch combines several selected examples into one training update#

A minibatch combines several selected examples into one training update. Averaging their gradients balances their contributions and can reduce sampling noise while remaining cheaper than a full-dataset calculation. Every gradient must be evaluated at the same current parameter.

A minibatch gradient is the average of its per-example gradients. Add matching coordinates, then divide by the number of selected examples.

\bar g=\frac1b\sum_{i=1}^{b}g_i

Gradients (2,−1), (5,2) and (2,5) sum to (9,6). Their batch average is (3,2).

Figure 1Three gradient vectors share an origin and equal coordinate units: g₁=(2,−1), g₂=(5,2), g₃=(2,5). Their mean is (3,2), the centroid of their endpoint triangle. The highlighted arrow is this average gradient, not a parameter update.

Link to this figure ↗Download SVG Download PNG

For gradients $(2,-1)$ , $(5,2)$ , and $(2,5)$ , sum first coordinates to get 9 and second coordinates to get 6. Divide both sums by the batch size 3, giving $(3,2)$ . There are three examples, not six: the number of vector entries is not the averaging denominator.

Check your reasoning

Example gradients: $(4, 2)$ , $(2, 6)$ . Find their average.

A $(3, 6)$
B $(3, 4)$
C $(4.5, 4)$

Show answer and explanation

$(3, 4)$

Sum each coordinate and divide by 2: $(3, 4)$ .

02If a batch selects only some rows, average exactly those rows#

If a batch selects only some rows, average exactly those rows. A repeated selected row contributes once per occurrence.

The result is a vector with the same shape as each gradient; do not collapse coordinates into one scalar.

Check your reasoning

Gradients: A= $(2, 1)$ , B= $(6, 5)$ , C= $(4, -1)$ . Average selected rows A,B,B.

A $(14/3,11/3)$
B $(14/3,11/2)$
C $(7,11/3)$

Show answer and explanation

$(14/3,11/3)$

Count B twice: average $(14/3,11/3)$ .

03A gradient sum differs from a gradient average by the batch count#

A gradient sum differs from a gradient average by the batch count. The selected objective’s normalization determines which is required.

Using only the last example discards the other selected contributions.

Check your reasoning

Example gradients: $(2, 5)$ , $(4, 3)$ . A draft reports their sum. Give the correct average.

A $(6, 8)$
B $(4.5, 4)$
C $(3, 4)$

Show answer and explanation

$(3, 4)$

Sum each coordinate and divide by 2: $(3, 4)$ .

04A mean loss uses a mean gradient#

A mean loss uses a mean gradient. If your objective instead sums losses, summing gradients is appropriate and changes the effective step scale. Keep the normalization consistent when comparing updates across batch sizes.

Key takeaway

Sum matching coordinates, retain selected multiplicity and divide by the actual batch size.

Average a minibatch's per-example gradients.

Sources & further reading

[1]
D2L §12.5.2, equations12.5.1–12.5.2 ↗D2L §12.5.2, equations12.5.1–12.5.2 · Article

01A minibatch combines several selected examples into one training update#

02If a batch selects only some rows, average exactly those rows#

03A gradient sum differs from a gradient average by the batch count#

04A mean loss uses a mean gradient#

Sum matching coordinates, retain selected multiplicity and divide by the actual batch size.

Sources & further reading

Reference this concept