Jacobians

01Organize several outputs and inputs#

A gradient collects coordinate rates for one scalar output. A vector-valued function has several output coordinates, each with its own dependence on the inputs. Its Jacobian puts all those first derivatives in a matrix.

We use one row per output and one column per input. If there are $n$ inputs and $m$ outputs, the Jacobian has shape $m\times n$.

Write $F_i$ for output $i$, and $x_j$ for input $j$. The entry in row $i$, column $j$ is

Read $J_F$ as “the Jacobian of $F$.” The subscript names the function. Writing $J_F(p)$ means evaluate all its entries at the input point $p$.

02Build the matrix one output at a time#

Consider $F(x,y)=(x^2+y,\ xy)$. Its first output is $F_1=x^2+y$. Differentiating that output with respect to $x$ gives $2x$; differentiating it with respect to $y$ gives $1$.

The first row is therefore $(2x,1)$. Both entries describe the same output, under two different input changes.

The second output is $F_2=xy$. Its $x$-partial is $y$, because $y$ stays fixed. Its $y$-partial is $x$, because $x$ stays fixed. This gives the second row $(y,x)$.

Read down a column to see how one input affects every output. Read across a row to see how every input affects one output.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

For a single scalar output, this convention produces a one-row Jacobian. Our gradient convention uses a column, so for a differentiable scalar function the Jacobian is the transpose of its gradient.

Neither convention changes any partial derivative. It changes how we arrange the entries so that matrix products have the intended meaning.

03Keep the output and input counts separate#

Now let $H(x,y)=(x+y,\ x^2,\ 2y)$. There are three outputs and two inputs, so we need three rows and two columns:

The second row's zero records that $x^2$ does not depend on $y$. The third row's zero records that $2y$ does not depend on $x$.

Zero entries are part of the map. Removing them would lose the alignment between derivatives and input coordinates.

A Jacobian need not be square. A model can turn two features into three outputs, or many parameters into one loss. Count outputs and inputs before calculating entries.

04Evaluate the derivative at a point#

Return to $F(x,y)=(x^2+y,xy)$ and choose $p=(1,3)$. Evaluating the function gives $F(p)=(4,3)$. Evaluating the Jacobian gives

The output is a pair of values; the Jacobian is a table of rates. In particular, the $3$ in the bottom-left position means the second output's local rate with respect to $x$ is three at this point.

When this function follows another function, its input may be an intermediate output rather than the original starting point. The notation in parentheses tells you where to evaluate.

For example, $J_F(G(p))$ says: calculate $G(p)$, then put those coordinates into the derivative formulas for $F$. It does not mean multiply $J_F$ by $G$ or by $p$.

05Use the Jacobian as a local linear map#

For a small input change $\Delta p$, a differentiable vector map satisfies the first-order approximation

The Jacobian maps an input change to an approximate output change. It does not replace the original nonlinear function for arbitrary inputs.

At $p=(1,3)$, choose $\Delta p=(0.01,-0.02)^T$. Our Jacobian predicts

The two contributions to the first output cancel to first order, while the second output is predicted to increase by $0.01$.

The actual output at $(1.01,2.98)$ is $(4.0001,3.0098)$. Compared with $F(p)=(4,3)$, the actual change is $(0.0001,0.0098)$, close to the prediction.

The remaining terms are nonlinear effects. This local-map interpretation explains why the chain rule multiplies Jacobians: a small change passes through one local map, then the next.

Further questions