Multivariable chain rule

01Follow a concrete composition#

A change in one input can reach an output through several intermediate coordinates. The multivariable chain rule accounts for all those paths. We will first follow actual values, then calculate the two local derivative maps.

Let the first function be $G(x,y)=(x+y,xy)$. It adds and multiplies the original inputs. Let the second be $F(u,v)=(u^2,u+v)$. It squares the first intermediate coordinate and adds the two.

Start at $p=(2,1)$. Here $p$ is simply a name for the input point. Apply $G$ to get the intermediate point $q$:

Then apply $F$ to those coordinates: $F(q)=(9,5)$. The notation $F\circ G$ means this composition: $G$ first, then $F$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

02Build each Jacobian where its function operates#

The diagram shows values, not derivatives. To describe small changes around them, build one Jacobian for each function. We use output rows and input columns.

For $G(x,y)=(x+y,xy)$, the two rows are $(1,1)$ and $(y,x)$. At the original point $p=(2,1)$:

Read $J_G(p)$ as “the Jacobian of $G$, evaluated at $p$.” The subscript $G$ names the function. The parentheses identify the point where its partial derivatives are evaluated. Neither symbol denotes an extra multiplication.

This matrix turns a small change in $(x,y)$ into a first-order change in $(u,v)$ near $p$.

For $F(u,v)=(u^2,u+v)$, the first row of derivatives is $(2u,0)$ and the second is $(1,1)$. This function received $q=(3,2)$, so its Jacobian is

The six comes from $2u$ at $u=3$. The zero records that $u^2$ has no dependence on $v$.

You will often see $J_F(G(p))$ instead of $J_F(q)$. They mean exactly the same thing because $q=G(p)$.

Evaluating the outer derivative at the original $p=(2,1)$ would put $4$ in its first entry instead of $6$. That is the derivative at an input the outer function did not receive.

03Compose the local maps in order#

Write $A=J_G(p)$ and $B=J_F(q)$ for a moment. A small original input change $\Delta p$ first becomes approximately $A\Delta p$. The next map turns that into $B(A\Delta p)$.

That is why the combined Jacobian is $BA$: the rightmost matrix acts first. In full notation, for differentiable maps,

Multiply the outer Jacobian by the inner one:

The first row comes from $6\cdot1+0\cdot1=6$ and $6\cdot1+0\cdot2=6$. The second comes from $1\cdot1+1\cdot1=2$ and $1\cdot1+1\cdot2=3$. Each entry adds contributions through both intermediate coordinates.

04Check the result and the evaluation point#

We can verify the composition without using the matrix rule. Substitute $G$ directly into $F$:

The first output has partial derivatives $2(x+y)$ and $2(x+y)$. The second has partial derivatives $1+y$ and $1+x$. At $(2,1)$, these give the same rows $(6,6)$ and $(2,3)$.

The original and intermediate points are both pairs, so either would fit into the outer formulas. Their equal dimensions do not make them interchangeable.

For a nonlinear outer function, moving its evaluation point can change the derivative. Always compute the forward intermediate values before substituting into the local Jacobians.

05Check dimensions and account for every path#

Reversing our two square matrices is possible arithmetically, but gives the wrong map:

Matching dimensions alone cannot establish the order. Follow the input change: inner derivative first, outer derivative second.

Rectangular maps make the dimensions more visible. If $G$ takes 2 inputs to 3 outputs, its Jacobian is $3\times2$. If $F$ takes those 3 coordinates to 2 outputs, its Jacobian is $2\times3$.

The product has shape $(2\times3)(3\times2)=2\times2$, matching two original inputs and two final outputs. Reversing these factors instead describes a $3\times3$ array, the wrong shape for this composition.

The same rule includes a scalar output reached through two paths. Let $u=x^2$, $v=3x$, and $L=uv$. At $x=2$, we have $u=4$ and $v=6$.

The outer derivative row is $(v,u)=(6,4)$. The inner derivative column is $(2x,3)^T=(4,3)^T$. Their product is $6(4)+4(3)=36$: one contribution through $u$, plus another through $v$.

Direct substitution gives $L=3x^3$, whose derivative at $2$ is also $36$. Dropping either path would miss part of that change.

This is the connection to backpropagation: forward values determine where local derivatives are evaluated, multiplication carries effects through operations, and addition collects contributions that meet at the same variable.

Further questions