Gradient clipping

01An unusually large gradient can propose a disruptive parameter update#

An unusually large gradient can propose a disruptive parameter update. Whole-vector clipping limits the gradient’s length while preserving its direction. It is a guard on the proposed gradient, separate from the optimizer’s learning rate.

Whole-vector clipping caps Euclidean length at T>0. For a nonzero gradient, use one common factor:

A zero vector stays zero; never divide by its norm.

Gradient (6,8) has norm 10. Threshold 5 gives common factor 0.5, so the clipped vector is (3,4).

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

For $g=(6,8)$, squared length is $36+64=100$, so length is 10. A threshold 5 gives scale factor $5/10=0.5$. Apply the same factor to both entries to get $(3,4)$, whose length is 5. If the learning rate is 0.1, the eventual plain-gradient step has length 0.5.

02When clipping precedes an optimizer step, first obtain the clipped vector#

When clipping precedes an optimizer step, first obtain the clipped vector. Then apply the learning rate to that result.

The threshold limits gradient length. It does not itself specify the parameter-step length, which also depends on the learning rate.

03Clipping each coordinate separately changes a different object and can change direction#

Clipping each coordinate separately changes a different object and can change direction. Whole-vector clipping uses one common nonnegative factor.

Vectors already at or below the threshold stay unchanged. Clipping should never enlarge them to reach the threshold.

04A closer look#

Clipping is used to control very large gradients during training, including in recurrent networks. It can limit an update’s size but cannot repair an incorrectly computed gradient. Componentwise clipping is a different operation and can change the direction.