Gradients

01Collect the coordinate rates#

A scalar function with several inputs has a partial derivative for each input. The gradient collects those partial derivatives in input order. It lets us keep the local effects of all coordinates together.

For $f(x,y)=x^2+3y$, the $x$-partial is $2x$ and the $y$-partial is $3$. We use column vectors:

The symbol $\nabla$ is called nabla. Read $\nabla f$ as “the gradient of $f$.” Its first entry belongs to the first input, $x$, and its second belongs to $y$.

At $p=(2,1)$, the function value is $f(p)=7$, while the gradient is $\nabla f(p)=(4,3)^T$. The superscript $T$ transposes the written row into a column. A value of seven and rates of four and three answer different questions.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

The diagram draws $(4,3)$ as an arrow from the origin to make its components visible on equal scales. It represents the gradient evaluated at $(2,1)$; its drawn starting point is not the evaluation point.

In an input-space plot, the same gradient direction could instead be attached at $(2,1)$. Separating the point where we evaluate from the vector we obtain prevents a common reading mistake.

02Evaluate every entry at the same point#

For a second function, $h(x,y)=x^2+3y^2$, both partial derivatives depend on the input:

At $(2,-1)$, the gradient is $(4,-6)^T$. The negative second entry says that a small positive change in $y$ decreases $h$ there, while a small positive change in $x$ increases it.

First build the symbolic gradient, then substitute the same point into every entry. Using an $x$-derivative at one point and a $y$-derivative at another would not give the gradient at either point.

Also keep the coordinate order. Swapping $(4,-6)^T$ to $(-6,4)^T$ attaches each rate to the wrong input.

03Predict a small change in the output#

The gradient becomes useful when several inputs move. For a differentiable function and a small input change $\Delta p$, the first-order prediction is

This dot product multiplies each coordinate's rate by its change, then adds the contributions. With column vectors, the transpose makes a row times a column, producing one scalar change.

Return to $f=x^2+3y$ at $(2,1)$, where the gradient is $(4,3)^T$. Increase $x$ by $0.01$ and $y$ by $0.02$.

The actual change is $f(2.01,1.02)-f(2,1)=0.1001$. The small difference comes from curvature, which the first-order prediction leaves out.

A zero partial derivative still occupies a coordinate slot. For $r(x,y)=x^2$, the gradient is $(2x,0)^T$, not just $2x$.

The zero says that this function has no dependence on $y$. For other functions, a zero derivative at one point may only describe that point; it does not establish independence everywhere.

04Follow the gradient as the input changes#

Some partial derivatives involve several coordinates. For $s(x,y)=x^2y$, treating the other coordinate as constant gives

At $(2,3)$, this is $(12,4)^T$. At $(1,3)$, it is $(6,1)^T$. Changing only $x$ changed both entries of the gradient.

The gradient is generally a function of position, not one permanently fixed vector. A gradient computed before a parameter update belongs to the old parameter state.

This is why optimization usually evaluates a new gradient after moving. Reusing the previous one forever follows information from a location the model has already left.

05Connect the gradient to optimization#

Under Euclidean distance, a nonzero gradient points in the direction of greatest first-order increase per unit of input movement. The negative gradient points toward greatest first-order decrease.

The qualification “first-order” matters. It describes sufficiently small movement near the evaluation point. A long step can pass through a minimum and increase the function.

For a scalar loss depending on a weight matrix $W$, we can collect the entrywise partial derivatives back into a matrix of the same shape:

This is the same organizing idea: one rate for every independent parameter. A matrix-shaped gradient of a scalar loss is different from the full derivative of a vector-valued function, which the Jacobian represents.

Further questions