Skip to content
Glacius
StatisticsConcept reference

Fitting regression

For a full-column-rank design X=QR, least-squares coefficients solve R beta = Q-transpose y.

2 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. A full-rank design has a unique least-squares coefficient vector
  3. Suppose R has rows (2,2),(0,2) and d=(6,4)
  4. A closer look
  5. Key takeaway
  6. Further questions
  7. Sources & further reading
  8. Concept connections
Fitting regression with QR: observed y is projected to Qᵀy, then a triangular solve with R produces coefficients β. The flow is a computation, with no distance scale.Fitting regression with QR: observed y is projected to Qᵀy, then a triangular solve with R produces coefficients β. The flow is a computation, with no distance scale.
Figure 1Fitting regression with QR: observed y is projected to Qᵀy, then a triangular solve with R produces coefficients β. The flow is a computation, with no distance scale.
Link to this figure ↗Download SVGDownload PNG

01A full-rank design has a unique least-squares coefficient vector#

A full-rank design has a unique least-squares coefficient vector. Use its supplied thin QR factorization: X=QRX=QR, with orthonormal columns in QQ and nonzero diagonal entries in RR.

To compute d=QTyd=Q^Ty, dot each column of QQ with yy. Let 2Q2Q have rows (1,−1)(1,-1); (1,−1)(1,-1); (1,1)(1,1); (1,1)(1,1) and y=(2,4,0,2)y=(2,4,0,2).

The first dot is (2+4+0+2)/2=4(2+4+0+2)/2=4. The second is (−2−4+0+2)/2=−2(-2-4+0+2)/2=-2. Thus d=(4,−2)d=(4,-2). Next solve Rβ=dR\beta=d.

Check your reasoning

Full rank. 2Q2Q rows (1,−1)(1,-1); (1,−1)(1,-1); (1,1)(1,1); (1,1)(1,1); y=(0,2,2,4)y=(0,2,2,4). RR rows (2,2)(2,2); (0,2)(0,2). Solve Rβ=QTyR\beta=Q^Ty.

  1. A(2,1)(2,1)
  2. B(1,1)(1,1)
  3. C(0,2)(0,2)
Show answer and explanation
(1,1)(1,1)

d=(4,2)d=(4,2); β=(1,1)\beta=(1,1).

02Suppose R has rows (2,2),(0,2) and d=(6,4)#

Suppose RR has rows (2,2),(0,2)(2,2),(0,2) and d=(6,4)d=(6,4). The last equation is 2β1=42\beta_1=4, so the slope is 22.

2β1=42β0+4=6\begin{aligned}2\beta_1&=4\\2\beta_0+4&=6\end{aligned}

Use that slope in the first equation: 2β0+2(2)=62\beta_0+2(2)=6. Subtract 44, then divide by 22. The intercept is 11, so the full result is (1,2)(1,2).

Check your reasoning

RR rows (2,2)(2,2); (0,2)(0,2), d=(10,6)d=(10,6). Diagonal-only answer (5,3)(5,3). Repair Rβ=dR\beta=d.

  1. A(2,3)(2,3)
  2. B(5,3)(5,3)
  3. C(−1,6)(-1,6)
Show answer and explanation
(2,3)(2,3)

d=(10,6)d=(10,6); β1=3\beta_1=3, β0=2\beta_0=2.

03A closer look#

Dividing each right-hand side by the matching diagonal is insufficient when RR has an off-diagonal term. Back substitution first removes the already-solved coefficient’s contribution.

Report coefficients in the design’s column order. A small training residual describes the fit to these observations; it does not establish accuracy on new observations.

Check your reasoning

Columns: intercept, x. RR rows (2,−2)(2,-2); (0,2)(0,2), QTy=(6,4)Q^Ty=(6,4). Give coefficients.

  1. A(3,2)(3,2)
  2. B(7,4)(7,4)
  3. C(5,2)(5,2)
Show answer and explanation
(5,2)(5,2)

d=(6,4)d=(6,4); β1=2\beta_1=2, β0=5\beta_0=5.

Key takeaway

Compute Q-transpose y, solve the triangular system backward, and report every coefficient in the design’s column order.

  • Obtain coefficients for a full-rank regression using a supplied QR least-squares procedure.

Further questions

What do observation-minus-prediction residuals tell us about this fit?
Subtract each fitted prediction from its observation to get a signed residual. These residuals describe the fit on the observed records; new-data accuracy needs separate evidence.

Sources & further reading

  1. [1]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Fitting regression.” Math behind ML. /learn/s-fit