Learning rates

01The learning rate turns a direction into a distance#

The learning rate turns a direction into a distance. Too small a rate can make progress slow; too large a rate can jump past a minimum and grow the error. A simple quadratic lets us calculate these outcomes exactly instead of relying on a sketch.

For a positive quadratic, the learning rate determines how each update scales the current position.

For f(x)=x², a rate of 0.75 takes x from 4 to −2. It crosses the minimum but cuts the loss from 16 to 4.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

For $f(x)=x^2/2$, the gradient is $x$, so one step gives $x_{new}=(1-\eta)x$. With $x=2$ and $\eta=0.5$, the new value is 1. With $\eta=1.5$, it is $-1$: the sign flips but the distance to zero shrinks. With $\eta=3$, it is $-4$ and the distance grows.

02A negative factor flips the sign#

A negative factor flips the sign. Its absolute value determines whether distance from zero shrinks, stays fixed or grows.

For a nonzero start, repeated fixed-rate steps converge to zero when the factor’s absolute value is below 1.

03Crossing the minimum is not the same as diverging#

Crossing the minimum is not the same as diverging. A factor between −1 and 0 alternates signs while shrinking the distance.

At factor −1 the loss stays fixed; a factor below −1 makes distance and loss grow.

04A closer look#

For this quadratic, improvement depends on the magnitude of the multiplier $1-\eta$, not just whether it is positive. More complicated losses have different curvature in different regions and directions. A step-size rule must be interpreted for the stated objective.