CalculusConcept reference

Multivariable chain rule

For differentiable vector maps in output-row convention, the composition Jacobian is the outer Jacobian at the inner output times the inner Jacobian at the original input.

2 min read1 figureSources & further reading

On this page 9 sections

Overview
Understand the idea
Let G(x,y)=(2x,y) and F(u,v)=(u^2,v)
A closer look
A parameter may influence several intermediate model values before they meet in the loss
Key takeaway
Further questions
Sources & further reading
Concept connections

01Understand the idea#

When several intermediate variables all depend on an input, that input can affect the final output along several paths. The multivariable chain rule adds the contributions from those paths. Multiplication carries a rate along one path; addition combines different paths.

For $F\circ G$ , the inner map $G$ acts first. The outer map $F$ receives $G(p)$ , so its derivative must be evaluated there.

Figure 1A schematic composition sends input p through G to G(p), then through F to F(G(p)). The corresponding derivative factors are J_G(p) for the first arrow and J_F(G(p)) for the second. Acting on an input change applies the inner factor first, so the matrix product is outer times inner.

Link to this figure ↗Download SVG Download PNG

For differentiable maps, multiply the outer Jacobian by the inner Jacobian. We use output rows and input columns.

\begin{gathered}J_{F\circ G}(p)\\=J_F(G(p))J_G(p)\end{gathered}

Let $u=x^2$ , $v=3x$ , and $y=uv$ . At $x=2$ , the forward values are $u=4,v=6$ . Through $u$ , the contribution is $(\partial y/\partial u)(du/dx)=6(4)=24$ . Through $v$ , it is $(\partial y/\partial v)(dv/dx)=4(3)=12$ . Adding gives $dy/dx=36$ .

Check your reasoning

$J_G(p)$ rows: $(1,0);(2,1)$ . $J_F(G(p))$ rows: $(3,0);(0,2)$ . Return the ordered rows of $J_{F\circ G}(p)$ .

A $(3,0);(4,2)$
B $(3,0);(6,2)$
C $(3,0);(0,2)$

Show answer and explanation

$(3,0);(4,2)$

$(2,1)$ : $0+4=4$ .

02Let G(x,y)=(2x,y) and F(u,v)=(u^2,v)#

Let $G(x,y)=(2x,y)$ and $F(u,v)=(u^2,v)$ . At $p=(1,2)$ , first find the intermediate input.

G(1,2)=(2,2)

For $G(x,y)=(2x,y)$ and $F(u,v)=(u^2,v)$ at $p=(1,2)$ , inner diagonal entries are 2,1. At $G(p)=(2,2)$ , outer entries are 4,1. Multiply to get 8,1.

J_{F\circ G}(1,2)=\begin{pmatrix}8&0\\0&1\end{pmatrix}

Check your reasoning

$G(x,y)=(x+y,y)$ , $F(u,v)=(u^2,v)$ . At $p=(1,2)$ , give $J_{F\circ G}(p)$ as ordered rows.

A $(2,2);(0,1)$
B $(6,6);(0,1)$
C $(6,1);(0,1)$

Show answer and explanation

$(6,6);(0,1)$

Outer entry: $2(3)=6$ .

03A closer look#

If $G$ has 2 inputs and 3 outputs, and $F$ has 3 inputs and 2 outputs, the Jacobians have shapes $3\times2$ and $2\times3$ . Outer times inner gives the required $2\times2$ result. Square factors still need the same order.

Check your reasoning

$J_G(p)$ rows: $(1,2);(0,1)$ . $J_F(G(p))$ rows: $(2,0);(1,1)$ . Repair reversed factors: give $J_{F\circ G}(p)$ as ordered rows.

A $(4,2);(1,1)$
B $(2,0);(0,1)$
C $(2,4);(1,3)$

Show answer and explanation

$(2,4);(1,3)$

$(2,2)$ : $2+1=3$ .

04A parameter may influence several intermediate model values before they meet in the loss#

A parameter may influence several intermediate model values before they meet in the loss. Omitting a path gives an incomplete gradient. When the same dependence is written using Jacobians, matrix multiplication performs these multiply-and-add operations with the dimensions kept in order.

Key takeaway

Evaluate the intermediate input, then multiply outer by inner.

Compute the derivative of a composed vector map using Jacobians.

Further questions

Why does the inner factor sit on the right?

Matrix products act on a vector from right to left. An input change first passes through the inner derivative, then through the outer derivative.

Sources & further reading

[1]
MIT 18.S096, Lecture 1–2: Derivatives as Linear Operators ↗MIT OpenCourseWare · Course

01Understand the idea#

02Let G(x,y)=(2x,y) and F(u,v)=(u^2,v)#

03A closer look#

04A parameter may influence several intermediate model values before they meet in the loss#

Evaluate the intermediate input, then multiply outer by inner.

Further questions

Sources & further reading

Reference this concept