Skip to content
Glacius
StatisticsConcept reference

Regression residuals

A residual is r=y minus fitted y-hat, in the same observation order. It need not equal the unobserved population error.

1 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. Match each observation to its fitted prediction
  3. An observation below its prediction has a negative residual
  4. Residuals use an estimated prediction hat y
  5. Key takeaway
  6. Further questions
  7. Sources & further reading
  8. Concept connections

01Match each observation to its fitted prediction#

Match each observation to its fitted prediction. Here the differences are 4−3=14-3=1, 1−2=−11-2=-1, and 3−3=03-3=0.

Records A,B,C have fitted values 3,2,3 and observations 4,1,3. At each categorical record position a hollow point is fitted and a filled point observed. Vertical arrows encode residual observed minus fitted: +1,−1,0. Horizontal spacing represents record order, not a predictor.Records A,B,C have fitted values 3,2,3 and observations 4,1,3. At each categorical record position a hollow point is fitted and a filled point observed. Vertical arrows encode residual observed minus fitted: +1,−1,0. Horizontal spacing represents record order, not a predictor.
Figure 1Records A,B,C have fitted values 3,2,3 and observations 4,1,3. At each categorical record position a hollow point is fitted and a filled point observed. Vertical arrows encode residual observed minus fitted: +1,−1,0. Horizontal spacing represents record order, not a predictor.
Link to this figure ↗Download SVGDownload PNG

A residual uses the observed response minus the fitted prediction. Keep the same record order in all three vectors.

r=y−y^r=y-\hat y
Check your reasoning

In record order, y=(1,4,2)y=(1,4,2), y^=(2,3,2)\hat y=(2,3,2). Give r=y−y^r=y-\hat y.

  1. A(1,−1,0)(1,-1,0)
  2. B(−1,1,0)(-1,1,0)
  3. C(1,1,0)(1,1,0)
Show answer and explanation
(−1,1,0)(-1,1,0)

Residuals: (−1,1,0)(-1,1,0).

02An observation below its prediction has a negative residual#

An observation below its prediction has a negative residual. Squaring or taking absolute values loses that sign; those are different quantities.

Check your reasoning

In record order, y=(6,0)y=(6,0), y^=(4,1)\hat y=(4,1). Wrong (−2,1)(-2,1) from y^−y\hat y-y. Repair rr.

  1. A(2,−1)(2,-1)
  2. B(−2,1)(-2,1)
  3. C(2,1)(2,1)
Show answer and explanation
(2,−1)(2,-1)

Residuals: (2,−1)(2,-1).

03Residuals use an estimated prediction hat y#

Residuals use an estimated prediction y^\hat y. The population error subtracts the true conditional mean. The two differences need not match.

For a fitted rule y^=2+x\hat y=2+x at x=3x=3, prediction is 55. If the observed response is 44, the residual is −1-1.

Check your reasoning

Fitted y^=1+2x\hat y=1+2x. In record order, x=(2)x=(2), y=(4)y=(4). Give r=y−y^r=y-\hat y.

Show answer and explanation
-1

y^=5\hat y=5; 4−(5)=−14-(5)=-1.

Key takeaway

Subtract the fitted prediction from the observation. Keep the sign and record order.

  • Compute residuals from fitted predictions and observations.

Further questions

How can a linear score produce a probability instead of an unrestricted prediction?
The logistic link p=1/(1+exp(-z)) maps any finite real score z to a number strictly between zero and one. Applying the link is separate from fitting the score model.

Sources & further reading

  1. [1]
  2. [2]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Regression residuals.” Math behind ML. /learn/s-residuals