Partial derivatives

01Change one input at a time#

A function can depend on several inputs even when its output is a single number. A partial derivative asks what happens locally when one input changes and the others stay fixed. This is how we isolate the effect of one model parameter on a loss.

We will use $f(x,y)=x^2y+3y$. At $(x,y)=(2,4)$, its value is $16+12=28$. That value tells us where the output is; a derivative tells us how it changes.

To study the effect of $x$ at this point, hold $y$ at $4$. The two-input function becomes the one-input slice

Differentiate this ordinary function of $x$: the slope is $8x$. At $x=2$, the slope is $16$. Increasing $x$ a little, while keeping $y=4$, increases the output by about sixteen times that change.

02Read the partial derivative#

We write the rate with respect to $x$ as $\partial f/\partial x$, or $f_x$. The symbol $\partial$ is read as “partial.” In this lesson, $f_x$ means differentiation, not an extra input to $f$.

Keeping $y$ symbolic but constant gives the same result:

Holding $y$ constant does not mean setting it to zero. In $x^2y$, it acts as a fixed coefficient, so differentiating with respect to $x$ leaves $2xy$. The separate term $3y$ has no changing $x$, so its derivative is zero.

Choose the variable first. Then identify which factors are coefficients and which expressions actually change.

03Take a different slice through the same point#

Now change $y$ while holding $x=2$. The slice becomes $f(2,y)=4y+3y=7y$, whose slope is $7$.

In symbolic form,

The same point therefore has an $x$-rate of $16$ and a $y$-rate of $7$. There is no contradiction: they describe different changes in the input.

The diagram uses a separate example, $f(x,y)=x^2+3y$, to show a slice in the input plane. Along its highlighted horizontal path, $y=2$ stays fixed and $x$ varies. The resulting one-variable function is $f(x,2)=x^2+6$.

The vertical axis shows the input $y$, not the function's output. The line marks which inputs we visit.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

Return to $f(x,y)=x^2y+3y$ at $(2,4)$. With a small change $\Delta x=0.01$ and no change in $y$, the derivative predicts

The actual new value is $f(2.01,4)=28.1604$. The prediction $28.16$ is close, but the derivative is a local linear approximation, not an exact finite-change rule.

04Differentiate through an inner expression#

Partial derivatives still use the ordinary rules of differentiation. For $g(x,y)=(x+3y)^2$, first recognize the outer square and the inner expression $u=x+3y$.

For the $y$-partial, the outer rate is $2u$ and the inner rate is $3$. Multiply them:

For the $x$-partial of the same function, the inner rate is $1$, because $3y$ stays fixed. This gives $g_x=2(x+3y)$.

The outer operation did not change; the coordinate we allowed to vary did. Omitting that inner rate is why two apparently similar partial derivatives can be confused.

05Evaluate and interpret the result#

Try a new context: a scalar cost $C(a,b)=a^2b+2b$. Its partial derivatives are $C_a=2ab$ and $C_b=a^2+2$.

At $(a,b)=(3,2)$, these are $12$ and $11$. A small increase in $a$ alone has a different effect from the same-sized increase in $b$ alone. Both rates were evaluated at the same starting state.

Differentiate before substituting all the coordinates. If you replace $a$ and $b$ by numbers at the beginning, $C(3,2)=22$ is just a number; it no longer records how the cost depends on either input.

You may substitute the coordinates being held fixed to make a slice, as we did earlier. Keep the coordinate you are differentiating with respect to variable until you have taken its derivative.

A partial derivative is one piece of local information. It does not by itself describe a change in which several coordinates move together. The gradient collects all these coordinate rates, and the multivariable chain rule combines them when inputs depend on other changing quantities.

Further questions