Skip to content
Glacius
StatisticsConcept reference

Linear regression models

A regression model writes Y=m(x)+epsilon, with zero mean noise conditional on each supplied predictor value.

2 min read1 figureSources & further reading
On this page 9 sections
  1. Overview
  2. Understand the idea
  3. For Y=1+2x+epsilon, the mean at x=2 is 5
  4. A model may use a supplied feature such as h(x)=x^2
  5. A closer look
  6. Key takeaway
  7. Further questions
  8. Sources & further reading
  9. Concept connections

01Understand the idea#

A linear regression model describes an output as an intercept plus weighted features, with a residual for what the model does not explain. The weights are shared across observations; feature values change from one observation to the next.

Fix the predictor at x=1x=1. This toy model makes Y=2,3,4Y=2,3,4 equally likely. Their mean is 33, even though an individual response can differ.

At predictor x=1, the conditional model assigns equal probability 1/3 to responses 2,3,4. Their conditional mean is 3 on the line m(x)=1+2x. The three signed noise values are -1,0,1. The x and y coordinates use the same 26 pixels per unit.At predictor x=1, the conditional model assigns equal probability 1/3 to responses 2,3,4. Their conditional mean is 3 on the line m(x)=1+2x. The three signed noise values are -1,0,1. The x and y coordinates use the same 26 pixels per unit.
Figure 1At predictor x=1, the conditional model assigns equal probability 1/3 to responses 2,3,4. Their conditional mean is 3 on the line m(x)=1+2x. The three signed noise values are -1,0,1. The x and y coordinates use the same 26 pixels per unit.
Link to this figure ↗Download SVGDownload PNG

Write the response as a conditional mean plus noise. The noise must average to zero at each predictor value.

Y=m(x)+ϵY=m(x)+\epsilon
E[ϵ∣x]=0E[\epsilon\mid x]=0

With intercept 2 and weights (3,−1)(3,-1), observation (4,5)(4,5) has fitted value 2+3(4)−1(5)=92+3(4)-1(5)=9. If the observed target is 11, its residual under the convention observed minus fitted is 2. The fitted value, coefficient vector, and residual are different objects.

Check your reasoning

Y=−1+3x+ϵY=-1+3x+\epsilon, E[ϵ∣x]=0E[\epsilon\mid x]=0. Give m(x)=E[Y∣x]m(x)=E[Y\mid x].

  1. Am(x)=−1+3x+ϵm(x)=-1+3x+\epsilon
  2. Bm(x)=−1+3xm(x)=-1+3x
  3. Cm(x)=3xm(x)=3x
Show answer and explanation
m(x)=−1+3xm(x)=-1+3x

m(x)=−1+3xm(x)=-1+3x; the noise averages to zero.

02For Y=1+2x+epsilon, the mean at x=2 is 5#

For Y=1+2x+ϵY=1+2x+\epsilon, the mean at x=2x=2 is 55. A response of 44 has noise −1-1. The mean describes the conditional distribution; it does not force each response to equal 55.

Check your reasoning

Y=5−x+ϵY=5-x+\epsilon, E[ϵ∣x]=0E[\epsilon\mid x]=0. A learner calls 5−x+ϵ5-x+\epsilon the mean. Repair m(x)m(x).

  1. Am(x)=5−xm(x)=5-x
  2. Bm(x)=5−x+ϵm(x)=5-x+\epsilon
  3. Cm(x)=−xm(x)=-x
Show answer and explanation
m(x)=5−xm(x)=5-x

m(x)=5−xm(x)=5-x; the noise averages to zero.

03A model may use a supplied feature such as h(x)=x^2#

A model may use a supplied feature such as h(x)=x2h(x)=x^2. The mean 3+2h(x)=3+2x23+2h(x)=3+2x^2 is still linear in its coefficients. The transformation is fixed; the additive noise stays separate.

Check your reasoning

Feature h(x)=x2h(x)=x^2. Offset 11, coefficient −2-2; variable noise with E[ϵ∣x]=0E[\epsilon\mid x]=0. Choose YY.

  1. AY=1−2x+ϵY=1-2x+\epsilon
  2. BY=1−2x2Y=1-2x^2
  3. CY=1−2x2+ϵY=1-2x^2+\epsilon
Show answer and explanation
Y=1−2x2+ϵY=1-2x^2+\epsilon

m(x)=1−2x2m(x)=1-2x^2.

04A closer look#

A model can use transformed features, such as an input squared, while remaining linear in its coefficients. A fitted weight describes the specified model with other features held fixed; it is not automatically a causal effect. Model structure and interpretation should be chosen before fitting.

Key takeaway

Keep every supplied predictor term in the conditional mean. Add noise whose conditional mean is zero.

  • Specify a conditional mean model separating predictors from residual noise.

Further questions

How can observation rows encode these features and the intercept?
Put one observation on each row and one declared feature on each column. A column of ones carries the intercept coefficient.

Sources & further reading

  1. [1]
  2. [2]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Linear regression models.” Math behind ML. /learn/s-linear-model