Least-Squares Gradients

01Send residuals back to the parameters#

Consider the scalar objective $L(x)=\tfrac12\|Ax-b\|_2^2$, with fixed $A$ and $b$. The prediction is $Ax$ and the residual is $r=Ax-b$. The residual has one entry per output; the gradient needs one entry per parameter.

The half-squared norm is $\tfrac12\sum_i r_i^2$, so its gradient with respect to $r$ is $r$. The derivative of the residual map $Ax-b$ is $A$. Reverse-mode differentiation sends the output sensitivity back through its transpose.

02Follow the intermediate values#

Let $A$ have rows $(1,2),(2,0),(0,1)$, $b=(2,4,1)$, and $x=(0,0)$. Predictions are $(0,0,0)$ and residuals are $(-2,-4,-1)$.

The first gradient entry is $1(-2)+2(-4)+0(-1)=-10$. The second is $2(-2)+0(-4)+1(-1)=-5$. Thus the gradient is $(-10,-5)$, while the loss is $(4+16+1)/2=10.5$.

03Interpret the transpose#

For $A$ of shape $m\times d$, the residual is an $m$-vector. Multiplying by $A^T$, of shape $d\times m$, collects residual contributions into a $d$-vector. Each parameter receives contributions from every output it affects.

If the objective is the mean half-squared loss, divide both loss and gradient by $m$. The gradient above is for the stated sum. A different reduction changes its scale.

A zero gradient does not require a zero residual. It means the residual is perpendicular to every column of $A$. This connects the derivative calculation to the earlier geometry of the closest attainable output.