Matrix Product: Right-Input Gradient

01Accumulate across output rows#

Let $Z=AB$, with fixed $A$ of shape $m\times d$ and variable $B$ of shape $d\times k$. A scalar loss supplies $G=\nabla_ZL$, of shape $m\times k$. We want one derivative for every entry of $B$.

The entry $B_{rj}$ contributes to every output row in column $j$. Its contribution to output $Z_{ij}$ is scaled by $A_{ir}$. Summing the loss sensitivities gives

Collect those sums as

02Sum contributions from the batch#

Let $A$ have rows $(1,2),(2,0),(0,1)$ and let $G$ have rows $(2,-1),(1,3),(-2,4)$. Entry $(1,1)$ of $A^TG$ is $1(2)+2(1)+0(-2)=4$.

The other entries give gradient rows $(4,5),(2,2)$. The three output rows contribute to just two parameter rows. Shared parameters collect contributions from every example that uses them.

03Keep the incoming scale#

The shape is $(d\times m)(m\times k)=d\times k$, matching $B$. The transpose aligns the summation over output rows. Correct dimensions are a useful check, but the entrywise argument explains why this order is right.

If $G$ already represents an average loss, its averaging factor is already present in this sum. Dividing by $m$ again makes the update too small for the stated objective.

This rule treats the displayed $A$ as fixed for the partial derivative with respect to $B$. If the same underlying parameter appears elsewhere in a computation, its other derivative paths must also be accumulated.