Vector–Jacobian products

01Understand the idea#

A vector–Jacobian product combines a downstream sensitivity for each output into sensitivities for the inputs. It is the matrix form of a backward step. Instead of asking for the whole table of derivatives, it asks which weighted combination of those derivatives the current loss needs.

For $y=f(x)$, the Jacobian $J$ has one row per output and one column per input. Let $s$ be a column of output sensitivities. The input sensitivities form column $g$:

With $J=\begin{bmatrix}2&1\\0&3\end{bmatrix}$ and $s=(4,-1)^T$, each input collects contributions from both outputs: $2(4)+0(-1)=8$ and $1(4)+3(-1)=1$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

With output-row Jacobian rows $(2,1)$ and $(3,4)$, and upstream row sensitivities $(5,-1)$, weight the first Jacobian row by 5 and the second by $-1$. Their sum is $(10,5)+(-3,-4)=(7,1)$. The two resulting entries correspond to the two inputs.

02If J has shape m× n, then s has m entries and J^Ts has n entries#

If $J$ has shape $m\times n$, then $s$ has $m$ entries and $J^Ts$ has $n$ entries. With row-vector sensitivities, the same calculation is written $s^TJ$.

For $y_i=x_i^2$, the backward rule can act coordinate by coordinate: input sensitivity $i$ is $2x_i s_i$. No full diagonal matrix needs to be stored.

03J maps small input changes toward outputs#

$J$ maps small input changes toward outputs. Its transpose sends output sensitivities back toward inputs. For a square Jacobian, both products may fit the dimensions, so shape alone cannot choose the right direction.

The full backward product adds all output contributions for each input.

04A closer look#

Backpropagation can compute this product for each operation without storing a full model-wide Jacobian. Here the full small matrix is supplied so you can see the multiplication. Check the convention: an upstream row vector multiplies the output-row Jacobian on the left.