Matrix Product: Left-Input Gradient

01Send output sensitivities to the left input#

Let $Z=AB$, where $A$ is $m\times d$ and fixed $B$ is $d\times k$. Suppose a scalar loss supplies the incoming gradient $G=\nabla_ZL$, shaped $m\times k$. We want the gradient with respect to $A$.

Output $Z_{ij}=\sum_r A_{ir}B_{rj}$ depends on $A_{ir}$ with derivative $B_{rj}$. That input entry affects every output column in its row. Add their contributions:

This is a row-column product with a transpose:

02Collect all output columns#

Let $B$ have rows $(1,2,0),(-1,0,3)$ and let $G$ have rows $(2,1,-1),(0,3,2)$. For gradient entry $(1,1)$, compute $2(1)+1(2)-1(0)=4$.

For entry $(1,2)$, compute $2(-1)+1(0)-1(3)=-5$. The second row gives $6$ and $6$. The left-input gradient therefore has rows $(4,-5),(6,6)$.

03Match the input#

The product has shape $(m\times k)(k\times d)=m\times d$, matching $A$. An entrywise product cannot collect the contributions from several output columns.

Use $G$ exactly as supplied. If an earlier mean-loss derivative already divided by batch size, do not divide again here. This operation propagates sensitivities; it does not choose a new loss reduction.