Why gradient descent can oscillate

The gradient tells you which way a function increases locally. Taking a step in the opposite direction sounds like a reliable way to go downhill. So why can gradient descent bounce from one side of a minimum to the other—or get farther away?

A direction is not a step size. A locally useful direction can still carry you past the point you wanted to reach. A simple quadratic lets us calculate exactly when that happens.

A function we can see all the way through

Use $f(x) = x^2/2$. Its minimum is at $x = 0$, and its derivative is $f'(x) = x$. Start at $x_0 = 1$.

With a learning rate $\eta$, gradient descent takes the update:

$$x_{t+1} = x_t - \eta f'(x_t) = (1-\eta)x_t.$$

Each step multiplies the current value by the same number, $1-\eta$. Repeating the update gives $x_t = (1-\eta)^t$ for our starting point. We can understand the whole trajectory by looking at that multiplier.

Three rates, three stories

With $\eta = 0.5$, the multiplier is 0.5. The sequence is $1, 0.5, 0.25, 0.125, \ldots$. Each step closes half the remaining distance. The sign stays positive and the distance from zero gets smaller.

With $\eta = 1.5$, the multiplier is −0.5. Now the sequence is $1, -0.5, 0.25, -0.125, \ldots$. Each step crosses the minimum. It oscillates, but its distance from zero still halves. This trajectory converges too.

With $\eta = 2.1$, the multiplier is −1.1. The sequence begins $1, -1.1, 1.21, -1.331, \ldots$. It crosses zero and gets farther away every time. This one diverges.

The chart above plots the parameter $x$ against step number, rather than plotting the loss. That distinction matters: the parameter can oscillate even while the loss steadily falls.

The exact boundary in this example

For the distance to zero to shrink, the magnitude of the multiplier must be less than 1:

$$|1-\eta| < 1.$$

Solving gives $0 < \eta < 2$. Inside that interval, this quadratic converges from our starting point. At $\eta=2$, it keeps bouncing at a fixed distance; above 2, the distance grows.

The threshold 2 belongs to this particular function. It is not a universal learning-rate recommendation. If we change the function to $f(x)=cx^2/2$ for $c>0$, the derivative becomes $cx$, the multiplier becomes $1-\eta c$, and the convergence condition becomes $0<\eta<2/c$.

Greater curvature reduces the size of a stable step. This is one reason a learning rate that works on one objective can fail on another.

What carries over to machine learning?

Real training problems can have many parameters, different curvatures in different directions, and noisy gradient estimates. The simple multiplier above no longer describes the full process.

But it gives you a useful question to ask when a training run behaves strangely: is the update direction wrong, or is the step too large for the local geometry? Dive into Deep Learning develops this idea further with examples of gradient descent.

Changing the learning rate is an experiment to evaluate, not a diagnosis by itself. Inspect the loss, check gradient calculations when appropriate, and compare runs under controlled conditions. A noisy curve can have more than one explanation.

Predict the next step

For our original function, start again at $x=1$ and use $\eta=0.8$. Where does the first update land? Does the next one cross zero?

Show the answer

The multiplier is $1-0.8=0.2$. The first update lands at 0.2 and the second at 0.04. Neither crosses zero. Repeated updates approach zero from the positive side.

Continue with the step-size lesson, or revisit the gradient update to connect the formula to the direction of movement.

The figure’s downloadable CSV contains all nine plotted values for each rate, so you can reproduce the trajectories. These are exact-form calculations for an illustrative function, not measurements from a model-training benchmark.