Skip to content
Glacius
CalculusConcept reference

Gradients

The gradient of a scalar function is the vector of its coordinate partial derivatives in input order.

2 min read1 figureSources & further reading
On this page 9 sections
  1. Overview
  2. Understand the idea
  3. A gradient can depend on the input
  4. At a point, evaluate every partial at that same point
  5. A closer look
  6. Key takeaway
  7. Further questions
  8. Sources & further reading
  9. Concept connections

01Understand the idea#

The gradient collects all partial derivatives of a scalar function into one vector, in input order. It answers how the output responds to small changes in each input. In model training, the output is often a loss and the inputs are the adjustable weights.

The gradient collects coordinate partials into one vector. For f(x,y)=x2+3yf(x,y)=x^2+3y at (2,1)(2,1), its entries are 4 and 3.

For f(x,y)=x²+3y at (2,1), the gradient is (4,3). This free vector is anchored at the origin to show its components, not the parameter’s position. The x and y coordinate units are equal.For f(x,y)=x²+3y at (2,1), the gradient is (4,3). This free vector is anchored at the origin to show its components, not the parameter’s position. The x and y coordinate units are equal.
Figure 1For f(x,y)=x²+3y at (2,1), the gradient is (4,3). This free vector is anchored at the origin to show its components, not the parameter’s position. The x and y coordinate units are equal.
Link to this figure ↗Download SVGDownload PNG

For coordinates (x,y)(x,y), place fxf_x first and fyf_y second. Both entries describe rates of the same scalar function.

∇f=(fx,fy)\nabla f=(f_x,f_y)

For f(x,y)=x2+3y2f(x,y)=x^2+3y^2, differentiate with respect to xx while holding yy fixed to get 2x2x. Differentiate with respect to yy to get 6y6y. At (2,−1)(2,-1), the gradient is (4,−6)(4,-6). Evaluate both entries at the same point; the function value 7 is not a gradient entry.

Check your reasoning

Order (x,y)(x,y): fy=−2f_y=-2, fx=5f_x=5. Find ∇f\nabla f.

  1. A(−2,5)(-2,5)
  2. B33
  3. C(5,−2)(5,-2)
Show answer and explanation
(5,−2)(5,-2)

Put the x partial first.

02A gradient can depend on the input#

A gradient can depend on the input. For f(x,y)=x2yf(x,y)=x^2y, compute both partials, then assemble them in order.

∇f=(2xy,x2)\nabla f=(2xy,x^2)
Check your reasoning

Order (x,y,z)(x,y,z): fx=2f_x=2, fy=0f_y=0, fz=−1f_z=-1. A learner drops the zero. Repair ∇f\nabla f.

  1. A(2,0,−1)(2,0,-1)
  2. B(2,−1)(2,-1)
  3. C11
Show answer and explanation
(2,0,−1)(2,0,-1)

Keep the y slot, even when its value is zero.

03At a point, evaluate every partial at that same point#

At a point, evaluate every partial at that same point. For f(x,y)=x2yf(x,y)=x^2y and (x,y)=(2,3)(x,y)=(2,3), the gradient keeps both rates.

∇f(2,3)=(12,4)\nabla f(2,3)=(12,4)
Check your reasoning

Cost C(a,b)=a2+2abC(a,b)=a^2+2ab. At (a,b)=(1,3)(a,b)=(1,3), find ∇C\nabla C.

  1. A(2,8)(2,8)
  2. B(8,2)(8,2)
  3. C(7,7)(7,7)
Show answer and explanation
(8,2)(8,2)

Ca=8C_a=8 and Cb=2C_b=2.

04A closer look#

For differentiable functions with the usual Euclidean geometry, the gradient points in the direction of steepest local increase. Gradient descent moves oppositely. A finite step still needs a learning rate: local sensitivity alone does not guarantee improvement for an arbitrarily large move.

Key takeaway

One coordinate, one partial, one gradient slot.

  • Assemble a gradient from partial derivatives in coordinate order.

Further questions

Can I add the partials instead?
Adding them loses which rate belongs to which coordinate. A gradient retains every entry, including zeros.

Sources & further reading

  1. [1]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Gradients.” Math behind ML. /learn/c-gradient