Skip to content
Glacius
CalculusConcept reference

Directional derivatives

For a differentiable scalar function, the unit-direction derivative is its gradient dotted with the unit direction vector.

2 min read1 figureSources & further reading
On this page 9 sections
  1. Overview
  2. Partial derivatives test the coordinate axes
  3. For v=(3,4), the length is 5, so u=(0.6,0.8)
  4. A negative rate means the output locally decreases in that direction
  5. This calculation evaluates a proposed direction before an optimization step
  6. Key takeaway
  7. Further questions
  8. Sources & further reading
  9. Concept connections

01Partial derivatives test the coordinate axes#

Partial derivatives test the coordinate axes. A directional derivative asks about moving along another direction, such as changing two weights together. A unit direction makes the answer a rate per unit of distance moved, so directions can be compared fairly.

A direction vector can be long or short. For a rate per unit input distance, first turn a nonzero vector into a unit vector.

Equal-scale input plane, 28 pixels per unit. The long vector v=(3,4) and the short vector u=(0.6,0.8) start at the origin and point in the same direction. Their lengths are 5 and 1. Dividing v by 5 produces u. The output rate is not encoded by arrow length.Equal-scale input plane, 28 pixels per unit. The long vector v=(3,4) and the short vector u=(0.6,0.8) start at the origin and point in the same direction. Their lengths are 5 and 1. Dividing v by 5 produces u. The output rate is not encoded by arrow length.
Figure 1Equal-scale input plane, 28 pixels per unit. The long vector v=(3,4) and the short vector u=(0.6,0.8) start at the origin and point in the same direction. Their lengths are 5 and 1. Dividing v by 5 produces u. The output rate is not encoded by arrow length.
Link to this figure ↗Download SVGDownload PNG

For a differentiable scalar function, dot its gradient at the point with the unit direction. The result is a scalar rate.

Duf(p)=∇f(p)⋅uD_u f(p)=\nabla f(p)\cdot u

uu must have length 1.

With gradient (3,4)(3,4) and direction vector (0.6,0.8)(0.6,0.8), first check the direction has length 1. The rate is 3(0.6)+4(0.8)=53(0.6)+4(0.8)=5. Reversing the direction makes the rate −5-5. Using (3,4)(3,4) itself without normalizing gives 25, which also includes its fivefold movement speed.

Check your reasoning

Differentiable ff: ∇f(p)=(4,2)\nabla f(p)=(4,2), unit u=(0,1)u=(0,1). Find Duf(p)D_u f(p).

Show answer and explanation
2

4(0)+2(1)=24(0)+2(1)=2.

02For v=(3,4), the length is 5, so u=(0.6,0.8)#

For v=(3,4)v=(3,4), the length is 5, so u=(0.6,0.8)u=(0.6,0.8). If ∇f(p)=(5,0)\nabla f(p)=(5,0), the rate is 5(0.6)+0(0.8)=35(0.6)+0(0.8)=3. Divide the raw dot product 15 by length 5 for the unit-distance rate.

Check your reasoning

Differentiable ff: ∇f(p)=(3,0)\nabla f(p)=(3,0), v=(0,4)v=(0,4). Repair the claimed rate 4 per unit distance.

  1. A44
  2. B00
  3. C33
Show answer and explanation
00

u=(0,1)u=(0,1): rate 0.

03A negative rate means the output locally decreases in that direction#

A negative rate means the output locally decreases in that direction. If ∇f(p)=(2,0)\nabla f(p)=(2,0) and u=(−1,0)u=(-1,0), the rate is −2-2. A zero vector has no direction to normalize.

Check your reasoning

Cost C(a,b)=a2+bC(a,b)=a^2+b. At (a,b)=(1,2)(a,b)=(1,2), use direction v=(3,4)v=(3,4). Find the rate per unit distance.

Show answer and explanation
2

∇C=(2,1)\nabla C=(2,1); (6+4)/5=2(6+4)/5=2.

04This calculation evaluates a proposed direction before an optimization step#

This calculation evaluates a proposed direction before an optimization step. A negative rate predicts a decrease for a sufficiently small positive step under differentiability. It does not establish how far to move; the size of the step remains a separate choice.

Key takeaway

Normalize the direction, then dot with the gradient.

  • Compute a directional rate using a unit direction.

Further questions

Can I use a zero direction vector?
No. It has no direction and cannot be divided by its zero length.
Does doubling a direction double this rate?
A positive rescaling preserves the unit direction and its rate. Reversing direction reverses the sign.

Sources & further reading

  1. [1]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Directional derivatives.” Math behind ML. /learn/c-directional