Skip to content
Glacius
CalculusConcept reference

Matrix-Shaped Gradients

When a scalar loss depends on a matrix, its gradient has the same shape as that matrix.

2 min read0 figuresSources & further reading
On this page 7 sections
  1. Overview
  2. A derivative for each parameter entry
  3. Differentiate and place the entries
  4. Preserve shape and meaning
  5. Key takeaway
  6. Sources & further reading
  7. Concept connections

01A derivative for each parameter entry#

When a scalar loss depends on a matrix, its gradient has the same shape as that matrix. Entry (i,j)(i,j) holds the partial derivative with respect to parameter entry (i,j)(i,j).

(∇WL)ij=∂L∂Wij.(\nabla_W L)_{ij}=\frac{\partial L}{\partial W_{ij}}.

The output loss is one number. The gradient is an array because there is a separate sensitivity for every adjustable parameter. It is not the full Jacobian of a matrix-valued output.

02Differentiate and place the entries#

Let WW be 2×22\times2 and L(W)=W112+3W12−W21W22L(W)=W_{11}^2+3W_{12}-W_{21}W_{22}. The four partial derivatives, in row order, are 2W11,3,−W22,−W212W_{11},3,-W_{22},-W_{21}.

At WW with rows (2,1),(−1,4)(2,1),(-1,4), those partial derivatives give

∇WL=(43−41).\nabla_W L=\begin{pmatrix}4&3\\-4&1\end{pmatrix}.

For example, increasing W21W_{21} locally decreases the loss because its partial derivative is −4-4.

Check your reasoning

W has rows (1,2),(3,4). L=W₁₁²+2W₁₂−W₂₁. Gradient rows?

  1. A(2,2),(−1,0)
  2. B(1,2),(3,4)
  3. C(2,−1),(2,0)
Show answer and explanation
(2,2),(−1,0)

Place partial derivatives 2W₁₁,2,−1,0 in their original positions.

03Preserve shape and meaning#

A 3×23\times2 parameter matrix has a 3×23\times2 scalar-loss gradient, even if the model's prediction has a different shape. Check the variable named by the differentiation symbol.

If the loss does not depend on a particular entry, that entry's derivative is zero. If an entry appears in several terms, add all of its contributions before placing the derivative into the gradient.

Check your reasoning

A draft transposes the array of partial derivatives for a scalar loss of a rectangular W. What should it preserve?

  1. AThe shape of the output prediction
  2. BThe original entry positions and shape of W
  3. CA single row for every scalar loss
Show answer and explanation
The original entry positions and shape of W

Entrywise matrix gradients use the same layout as their variables.

A transposed layout can be a valid convention for a vector Jacobian, but here we use entrywise matrix gradients shaped like their variables. Keep that convention consistent throughout matrix backpropagation.

Key takeaway

Assemble the gradient of a scalar function of a real matrix from its entrywise partial derivatives.

  • Assemble the gradient of a scalar function of a real matrix from its entrywise partial derivatives.

Sources & further reading

  1. [1]
  2. [2]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Matrix-Shaped Gradients.” Math behind ML. /learn/c-matrix-gradient