Skip to content
Glacius
StatisticsConcept reference

Collinearity

Collinearity is dependence or near dependence among predictor columns that prevents stable separation of their coefficients.

2 min read1 figureSources & further reading
On this page 9 sections
  1. Overview
  2. Collinearity means that other feature columns explain one column exactly or approximately
  3. Exact dependence permits different coefficient vectors with identical fitted predictions
  4. For rows (1,1) and (2,2), coefficients (1,2) and (101,-98) both predict (3,6)
  5. A closer look
  6. Key takeaway
  7. Further questions
  8. Sources & further reading
  9. Concept connections

01Collinearity means that other feature columns explain one column exactly or approximately#

Collinearity means that other feature columns explain one column exactly or approximately. Recording a length in both meters and centimeters adds two columns but only one independent measurement. Regression may struggle to assign separate effects to their coefficients.

If one feature is xx and another is 2x2x, coefficients (2,0)(2,0) and (0,1)(0,1) produce the same prediction. The data cannot identify which allocation is correct. With nearly repeated columns, small data changes can lead to large coefficient changes even while fitted outputs remain close.

Identical predictor columns cannot distinguish separate coefficients. For rows (1,1)(1,1) and (2,2)(2,2), coefficient changes (100,−100)(100,-100) cancel in both predictions.

Design rows are (1,1) and (2,2). The coefficient change (100,−100) lies along beta1+beta2=0; both row dot products are zero. This is a coefficient-space direction, so moving along it leaves both fitted outputs unchanged.Design rows are (1,1) and (2,2). The coefficient change (100,−100) lies along beta1+beta2=0; both row dot products are zero. This is a coefficient-space direction, so moving along it leaves both fitted outputs unchanged.
Figure 1Design rows are (1,1) and (2,2). The coefficient change (100,−100) lies along beta1+beta2=0; both row dot products are zero. This is a coefficient-space direction, so moving along it leaves both fitted outputs unchanged.
Link to this figure ↗Download SVGDownload PNG

The prediction change is XΔβX\Delta\beta. For rows (1,1)(1,1) and (2,2.001)(2,2.001) with Δβ=(100,−100)\Delta\beta=(100,-100), the changes are 0 and 200−200.1=−0.1200-200.1=-0.1. Large opposing coefficients nearly cancel.

XΔβ=(0,−0.1)X\Delta\beta=(0,-0.1)

Use the complete singular-value list for the design matrix. A zero scale means an exact lost direction and deficient rank. A very small positive scale means that direction survives but is weak. For scales (10,0.01)(10,0.01), the ratio is 1000; calling the columns fully independent does not make the system well conditioned.

Check your reasoning

Design rows (1,1)(1,1); (3,3.01)(3,3.01). Coefficient shift (10,−10)(10,-10). Prediction shift?

  1. A(10,−10)(10,-10)
  2. B(0,−0.1)(0,-0.1)
  3. C(20,60.1)(20,60.1)
Show answer and explanation
(0,−0.1)(0,-0.1)

Dot products: (0,−0.1)(0,-0.1).

02Exact dependence permits different coefficient vectors with identical fitted predictions#

Exact dependence permits different coefficient vectors with identical fitted predictions. Near dependence can instead give a unique but sensitive fit: small response changes may require large coefficient changes. Check the design, not the coefficient sizes alone.

Check your reasoning

Design rows (1,2)(1,2); (2,4)(2,4). Coefficient shift (20,−10)(20,-10). Claim: prediction shift (20,−10)(20,-10). Repair.

  1. A(0,0)(0,0)
  2. B(20,−10)(20,-10)
  3. C(40,80)(40,80)
Show answer and explanation
(0,0)(0,0)

Dot products: (0,0)(0,0).

03For rows (1,1) and (2,2), coefficients (1,2) and (101,-98) both predict (3,6)#

For rows (1,1)(1,1) and (2,2)(2,2), coefficients (1,2)(1,2) and (101,−98)(101,-98) both predict (3,6)(3,6). That agreement holds on these rows. A new row (1,0)(1,0) gives 1 versus 101; do not extend the cancellation to arbitrary new data.

Check your reasoning

Design rows (1,1)(1,1); (2,2.02)(2,2.02). Two fits: (2,1)(2,1) and (12,−9)(12,-9). Give the largest absolute prediction difference.

Show answer and explanation
0.2

Prediction change (0,−0.2)(0,-0.2); largest absolute entry 0.20.2.

04A closer look#

Before interpreting a fitted coefficient, inspect whether another feature carries almost the same information. Removing redundancy or using a stated regularization method can change the fit. Collinearity by itself is not evidence that two measurements have a causal relationship.

Key takeaway

Multiply coefficient shifts by the design. Dependent or nearly dependent columns can make their prediction effect small.

  • Diagnose coefficient instability caused by dependent or nearly dependent predictors.

Further questions

Does stable prediction make each coefficient trustworthy?
No. Different coefficient vectors may yield similar predictions on the observed design. Interpreting individual coefficients requires attention to that instability.

Sources & further reading

  1. [1]
  2. [2]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Collinearity.” Math behind ML. /learn/s-collinearity