Local Change from a Matrix Gradient

01Predict a scalar change from a matrix change#

A scalar-loss gradient has one sensitivity per matrix entry. To estimate the effect of a small matrix perturbation, multiply matching sensitivities and changes, then add them all.

This is the same first-order reasoning as a gradient dot product for a vector. Keeping the matrix layout makes corresponding parameter positions visible.

02Combine signed contributions#

Suppose a gradient has rows $(2,-1),(0,3)$ and a proposed perturbation has rows $(0.1,0.2),(-0.1,-0.1)$. The four contributions are $0.2,-0.2,0,-0.3$.

Their sum is $-0.3$, so the first-order model predicts a loss decrease of about $0.3$. If the original loss is $5$, its estimated new value is $4.7$. The change and the new value are different answers.

03An approximation to finite changes#

A zero inner product means no first-order change; higher-order change may remain. For $L(W)=\|W\|_F^2/2$ at the zero matrix, the gradient is zero. A nonzero finite perturbation still increases the loss by half its squared norm.

For an affine scalar function of the entries, this calculation is exact. For a nonlinear function, smaller perturbations make the local interpretation more useful, but a large step needs its own evaluation.

An entrywise product alone is an array of contributions. The final sum is necessary because the loss change is a scalar. A row-by-column matrix product generally has the wrong shape and meaning here.