Squared Frobenius-Loss Gradients

01A mean squared loss for matrix outputs#

Let $Z$ contain model outputs and $Y$ contain fixed targets of the same shape. The loss $\|Z-Y\|_F^2/(2m)$ squares every entrywise error, sums them, and divides by a fixed positive scalar $2m$.

In a batch, $m$ often counts examples. It need not count all scalar output entries. Read the stated loss reduction before differentiating.

02One entry at a time#

Entry $Z_{ij}$ appears in the term $(Z_{ij}-Y_{ij})^2/(2m)$. The power and chain rules give $(Z_{ij}-Y_{ij})/m$. Placing these partials into their original positions gives

Suppose $Z$ has rows $(3,0),(1,4)$ and $Y$ has rows $(1,1),(1,2)$, with $m=2$. The residual rows are $(2,-1),(0,2)$. The loss is $(4+1+0+4)/4=2.25$.

The gradient rows are $(1,-0.5),(0,1)$. Each entry indicates the local sensitivity to its matching prediction. A prediction already equal to its target contributes a zero gradient entry.

03Average exactly once#

Dividing by four output entries instead of two examples would give a different objective and gradient. Both conventions can be useful, but they cannot be mixed halfway through a calculation.

The norm is squared. Differentiating the unsquared Frobenius norm would be a different operation. This gradient is with respect to predictions $Z$; reaching an earlier weight matrix requires another chain-rule step.