Linear algebraConcept reference

Least-squares geometry

A least-squares output is the attainable output that minimizes the sum of squared residual components relative to a target.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
Real measurements rarely lie exactly on a model’s predicted pattern
For target b=(2,4) and output p=(3,3), the residual is r=b-p=(-1,1)
An input coefficient t may produce output (2t,t)
A closer look
Key takeaway
Sources & further reading
Concept connections

Figure 1Equal scale of 30 pixels per data unit in both directions, with origin at pixel (45,170). Attainable outputs form the whole line y=x. The target b=(2,4) is off that line; its nearest output p=(3,3) lies on it. The residual arrow b−p=(-1,1) runs from p to b, perpendicular to direction (1,1). A right-angle marker shows this perpendicularity. The x and y axes are context only.

Link to this figure ↗Download SVG Download PNG

01Real measurements rarely lie exactly on a model’s predicted pattern#

Real measurements rarely lie exactly on a model’s predicted pattern. Least squares chooses the model output whose squared differences from the measurements have the smallest sum. It gives a precise meaning to “best fit” even when the equations have no exact solution.

The matrix $A$ describes allowed output patterns, $x$ contains their weights, and $b$ is the target. The residual $b-Ax$ is one error per measurement. Squaring avoids sign cancellation and gives larger errors more weight.

In $Ax=b$ , the vector $Ax$ is an attainable output. Suppose a model can output $(t,t)$ for any real $t$ , but the target is $b=(2,4)$ . Exact equality would require $t=2$ and $t=4$ at once.

Least squares chooses the attainable output closest to the target.

For target $b=(2,4)$ , output $p=(3,3)$ leaves residual $(-1,1)$ . Square both entries and add:

E=(-1)^2+1^2=2

Output $(2,2)$ gives squared error $4$ . Smaller squared distance means closer.

Check your reasoning

Outputs are $y=(t,t)$ for real $t$ . Target $b=(0,2)$ . Closest output?

A $(0,0)$
B $(1,1)$
C $(-1,1)$

Show answer and explanation

$(1,1)$

$b-y=(-1,1)$ is perpendicular to $(1,1)$ .

02For target b=(2,4) and output p=(3,3), the residual is r=b-p=(-1,1)#

For target $b=(2,4)$ and output $p=(3,3)$ , the residual is $r=b-p=(-1,1)$ . It is perpendicular to the output direction $(1,1)$ :

r\cdot(1,1)=0

In any linear output subspace, the closest output has a perpendicular residual.

Let $b=(2,4)$ and output $y=(3+h,3+h)$ . Its residual is $(-1-h,1-h)$ . Expanding and adding the squares cancels the linear terms:

E=2+2h^2

Since $h^2\ge0$ , output $(3,3)$ minimizes the error at $h=0$ .

Check your reasoning

Two channels output $(t,-t)$ for real $t$ . Target $(4,0)$ . Minimum sum of squared errors?

Show answer and explanation

Closest output $(2,-2)$ ; error $2^2+2^2=8$ .

03An input coefficient t may produce output (2t,t)#

An input coefficient $t$ may produce output $(2t,t)$ . For target $(0,5)$ , the closest output is $p=(2,1)$ : residual $(-2,4)$ is perpendicular to $(2,1)$ .

The coefficient is $t=1$ ; the output is the pair $(2,1)$ . Inconsistent targets leave nonzero error even at the closest output.

Check your reasoning

Outputs: $y=(t,0)$ , real $t$ . Target $b=(3,4)$ . Claim: exact fit. Closest output?

A $(3,0)$
B $(3,4)$
C $(0,0)$

Show answer and explanation

$(3,0)$

$(3,0)$ is attainable; its error is vertical.

04A closer look#

For repeated measurements 2 and 4 that must share one fitted constant, the prediction $(3,3)$ balances errors $(-1,1)$ . Their sum is zero, but the squared error is 2. This distinction is why a model can be the best available fit without matching every observation exactly.

Key takeaway

Choose an attainable output whose residual is perpendicular to the output subspace.

Identify the closest attainable output in an inconsistent linear system.

Sources & further reading

[1]
Margalit and Rabinoff: Interactive Linear Algebra, 6.5 The Method of Least Squares ↗Interactive Linear Algebra · Book
[2]
Margalit and Rabinoff: Interactive Linear Algebra, 6.3 Orthogonal Projection ↗Interactive Linear Algebra · Book

01Real measurements rarely lie exactly on a model’s predicted pattern#

02For target b=(2,4) and output p=(3,3), the residual is r=b-p=(-1,1)#

03An input coefficient t may produce output (2t,t)#

04A closer look#

Choose an attainable output whose residual is perpendicular to the output subspace.

Sources & further reading

Reference this concept