Skip to content
Glacius
Linear algebraConcept reference

Condition numbers

For a full-rank square system, the 2-norm condition number is the ratio of its largest to smallest singular value.

2 min read1 figureSources & further reading
On this page 9 sections
  1. Overview
  2. Understand the idea
  3. A closer look
  4. Uniform scaling does not change the ratio
  5. A closer look
  6. Key takeaway
  7. Further questions
  8. Sources & further reading
  9. Concept connections

01Understand the idea#

A condition number describes sensitivity: how much an output or solution can change relative to a small relative change in the data. A system can have a unique solution and still be difficult to solve accurately when its input measurements are uncertain.

For an invertible square matrix, a large singular value stretches one direction strongly while a small one barely changes another. Solving the system reverses those stretches. The weak direction then requires a large amplification, making uncertainty in that direction especially consequential.

A unique solution can still react strongly to small data changes. For a full-rank square system, the 2-norm condition number compares its largest and smallest singular values.

κ2=σmax⁡σmin⁡\kappa_2=\frac{\sigma_{\max}}{\sigma_{\min}}

Singular scales 8 and 0.5 give a ratio of 16. The system responds unevenly across directions: one is scaled sixteen times as much as the other.

Largest singular scale 8 has a 160px horizontal bar. Smallest scale 0.5 has a 10px bar. Their ratio is 16.Largest singular scale 8 has a 160px horizontal bar. Smallest scale 0.5 has a 10px bar. Their ratio is 16.
Figure 1Largest singular scale 8 has a 160px horizontal bar. Smallest scale 0.5 has a 10px bar. Their ratio is 16.
Link to this figure ↗Download SVGDownload PNG

For singular values (8,2,0.5)(8,2,0.5), identify the largest and smallest positive scales first. Their ratio is 8/0.5=168/0.5=16. The middle value does not enter this ratio. Multiplying the whole matrix by 3 makes the extremes 24 and 1.5, whose ratio is still 16. Overall size and relative sensitivity are different quantities.

Check your reasoning

Full-rank square system. Singular values (9,3)(9,3). Give κ2\kappa_2.

Show answer and explanation
3

9/3=39/3=3.

02A closer look#

A larger condition number means greater possible relative sensitivity in the inverse problem. It is a worst-case measure: a condition number of 16 does not mean every data change produces sixteen times the relative solution change.

Check your reasoning

Full-rank square system, scales (5,1)(5,1). Claim: unique solution means κ2=1\kappa_2=1. Repair.

  1. A1
  2. B5
  3. C0.2
Show answer and explanation
5

Extreme ratio: 5/1=55/1=5.

03Uniform scaling does not change the ratio#

Uniform scaling does not change the ratio. Scaling singular values (6,2)(6,2) to (60,20)(60,20) leaves the condition number at 3. Compare ratios, not the biggest or smallest value by itself.

6/2=60/20=36/2=60/20=3
Check your reasoning

Full-rank square inverse systems. Singular values: A (12,3)(12,3); B (4,1)(4,1). Compare worst-case relative sensitivity.

  1. AEqual
  2. BA
  3. CB
Show answer and explanation
Equal

κA=4\kappa_A=4; κB=4\kappa_B=4.

04A closer look#

In regression, nearly redundant features can make fitted coefficients very sensitive even when predictions change little. Conditioning helps explain that instability. The condition number is a worst-case amplification bound under the stated norm; it does not say every dataset actually suffers that much error.

Key takeaway

Largest divided by smallest measures possible relative sensitivity. Uniform scaling preserves this ratio.

  • Interpret the ratio of largest to smallest nonzero singular values for a full-rank square system.

Further questions

How can predictors make fitted coefficients unstable?
Dependent or nearly dependent predictor columns can let very different coefficients produce the same or nearly the same predictions. Conditioning quantifies this sensitivity in a full-rank setting.

Sources & further reading

  1. [1]
  2. [2]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Condition numbers.” Math behind ML. /learn/la-conditioning