Least-squares geometry

01Choose an attainable output#

A linear model can produce outputs $Ax$ by changing its coefficients $x$. Measurements $b$ may fall outside that set. Least squares selects the attainable output with the smallest sum of squared errors.

Suppose the allowed outputs are $(t,t)$ and the target is $(2,4)$. Matching both coordinates exactly would require $t=2$ and $t=4$ simultaneously. The target is inconsistent with the model, so some error remains.

Output $(3,3)$ leaves residual $b-Ax=(-1,1)$. Its squared error is $(-1)^2+1^2=2$. Output $(2,2)$ leaves $(0,2)$ and squared error $4$. Squaring keeps opposite errors from cancelling.

02Why the closest point wins#

Every allowed displacement from $(3,3)$ has the form $(h,h)$. Its dot product with the residual $(-1,1)$ is zero. The residual is perpendicular to the directions in which the model can move.

Moving to $(3+h,3+h)$ gives squared errors $(1+h)^2$ and $(1-h)^2$. Their linear terms cancel when added:

The extra term is nonnegative. No allowed move improves on $(3,3)$.

The same reasoning works for a linear output subspace. A perpendicular residual adds no cross term when the squared distance is expanded. Moving away adds the squared length of the allowed displacement.

03Coefficients and predictions differ#

For outputs $(2t,t)$ and target $(0,5)$, the closest output is $(2,1)$. Its residual $(-2,4)$ is perpendicular to direction $(2,1)$ because $-4+4=0$.

The fitted coefficient is $t=1$; the fitted output is the pair $(2,1)$. Reporting a coefficient when asked for an output confuses two different objects. The nonzero residual does not make this fit incorrect.

If the target already lies in the attainable subspace, its closest output is itself and the minimum error is zero. Otherwise, the closest output still exists in this finite-dimensional linear setting. Whether the coefficients producing it are unique is a separate question.

In ML, this geometry explains the best training fit within a chosen linear model. It does not establish performance on new observations. Computing coefficients efficiently and evaluating generalization have their own lessons.