Descent directions

A negative gradient-direction dot product identifies first-order descent at a differentiable interior point.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
Understand the idea
In a local linear model, the variable coefficients are the gradient components
Zero dot product makes this directional test inconclusive
A closer look
Key takeaway
Sources & further reading
Concept connections

01Understand the idea#

A descent direction is a proposed movement that decreases a differentiable objective locally. The gradient tells you how the objective responds to a small move. Taking its dot product with the proposed direction combines the effect of changing all coordinates together.

A direction’s first-order change is the gradient dot product with that direction. At a differentiable interior point:

\Delta f\approx\eta\,\nabla f^T d

For positive step size η, a negative dot product gives descent when the step is sufficiently small.

With gradient (3,−2) and direction (−1,2), the products sum to −7.

Figure 1Gradient g=(3,−2) and direction d=(−1,2) have dot product −7 and form an obtuse angle. For a differentiable function at an interior point, sufficiently small positive movement along d decreases the objective. Equal coordinate units preserve the angle.

For gradient $g=(2,-1)$ and direction $d=(-1,3)$ , $g\cdot d=-2-3=-5$ . The negative value predicts a decrease for a sufficiently small positive step. Reversing $d$ gives a positive dot product and predicts an increase. A zero dot product leaves the first-order test inconclusive.

Check your reasoning

At a smooth interior point, gradient $g=(4, -2)$ , direction $d=(-1, -1)$ . Compute $g^T d$ .

Show answer and explanation

-2

4×(-1)+(-2)×(-1)=-2.

02In a local linear model, the variable coefficients are the gradient components#

In a local linear model, the variable coefficients are the gradient components. Its constant term contributes no change.

Pair each coefficient with the matching direction component, then sum the products.

Check your reasoning

Local linear model: $m(h)=5+2h_1+4h_2$ . For $d=(1, -1)$ , find its change per unit step.

Show answer and explanation

-2

2×(1)+(4)×(-1)=-2.

03Zero dot product makes this directional test inconclusive#

Zero dot product makes this directional test inconclusive. Curvature may still raise or lower the objective along that direction.

A descending direction guarantees improvement only for sufficiently small positive steps. A large step can overshoot.

Check your reasoning

At a differentiable interior point, $g=(2,-1)$ , $d=(-1,1)$ . Draft: no decrease. For positive steps?

AAll steps decrease.
BAlready at a minimum.
CSmall steps decrease.

Show answer and explanation

Small steps decrease.

Dot product: −3.

04A closer look#

This test helps decide whether a proposed update points downhill before choosing a step length. It does not say every distance along that direction is safe. Curvature can make a large step increase the objective even when its initial direction is downhill.

Key takeaway

Compute the dot product: negative gives small-step descent, positive gives increase and zero is inconclusive.

Check whether a direction has a negative first-order objective change.

Sources & further reading

[1]
Dive into Deep Learning §12.3.2 ↗Dive into Deep Learning §12.3.2 · Article

01Understand the idea#

02In a local linear model, the variable coefficients are the gradient components#

03Zero dot product makes this directional test inconclusive#

04A closer look#

Compute the dot product: negative gives small-step descent, positive gives increase and zero is inconclusive.

Sources & further reading

Reference this concept