Skip to content
Glacius
ProbabilityConcept reference

Covariance matrices

A covariance matrix places the covariance of coordinates i and j in entry i,j, with variances on the diagonal.

2 min read1 figureSources & further reading
On this page 9 sections
  1. Overview
  2. A covariance matrix stores the pairwise variation of a vector’s components
  3. Covariance is symmetric: swapping the two variable names does not change it
  4. Uniform vectors (X,Y)=(1,2),(3,6) have mean (2,4)
  5. PCA uses this joint variation to find directions of spread
  6. Key takeaway
  7. Further questions
  8. Sources & further reading
  9. Concept connections

01A covariance matrix stores the pairwise variation of a vector’s components#

A covariance matrix stores the pairwise variation of a vector’s components. Diagonal entries describe each feature’s own variance; off-diagonal entries describe pairs. It is a compact description of how a multifeature cloud spreads around its mean.

A covariance matrix keeps every coordinate pair in a fixed order. Diagonal entries are variances; off-diagonal entries are covariances.

A schematic covariance matrix for vector order (X,Y,Z) has ordered rows (4,1,-1), (1,9,2), (-1,2,4). The diagonal entries are variances 4,9,4. Matching off-diagonal pairs both contain the same covariance. Row and column labels preserve coordinate order.A schematic covariance matrix for vector order (X,Y,Z) has ordered rows (4,1,-1), (1,9,2), (-1,2,4). The diagonal entries are variances 4,9,4. Matching off-diagonal pairs both contain the same covariance. Row and column labels preserve coordinate order.
Figure 1A schematic covariance matrix for vector order (X,Y,Z) has ordered rows (4,1,-1), (1,9,2), (-1,2,4). The diagonal entries are variances 4,9,4. Matching off-diagonal pairs both contain the same covariance. Row and column labels preserve coordinate order.
Link to this figure ↗Download SVGDownload PNG

For order (X,Y)(X,Y), variances 4 and 9 and covariance −2-2, place 4 and 9 on the diagonal. Both off-diagonal entries are −2-2.

Σ=(4−2−29)\Sigma=\begin{pmatrix}4&-2\\-2&9\end{pmatrix}

For two features with variances 4 and 9 and covariance −2-2, the matrix has rows (4,−2)(4,-2) and (−2,9)(-2,9). Covariance is symmetric, so both off-diagonal entries match. The standard deviations are 2 and 3; putting those on the diagonal would confuse spread units with squared spread.

Check your reasoning

Order (X,Y)(X,Y). Variances (1,9)(1,9). Covariances: XY=2XY=2. Give covariance rows.

  1. A(1,−2);(2,9)(1,-2);(2,9)
  2. B(1,2);(2,9)(1,2);(2,9)
  3. C(9,2);(2,1)(9,2);(2,1)
Show answer and explanation
(1,2);(2,9)(1,2);(2,9)

Diagonal (1,9)(1,9); mirror 22.

02Covariance is symmetric: swapping the two variable names does not change it#

Covariance is symmetric: swapping the two variable names does not change it. Diagonal entries are variances and cannot be negative. A negative off-diagonal entry is allowed.

Check your reasoning

Order (X,Y)(X,Y). Variances (4,1)(4,1). Covariances: XY=−1XY=-1. Fix opposite signs. Give rows.

  1. A(4,1);(−1,1)(4,1);(-1,1)
  2. B(1,−1);(−1,4)(1,-1);(-1,4)
  3. C(4,−1);(−1,1)(4,-1);(-1,1)
Show answer and explanation
(4,−1);(−1,1)(4,-1);(-1,1)

Diagonal (4,1)(4,1); mirror −1-1.

03Uniform vectors (X,Y)=(1,2),(3,6) have mean (2,4)#

Uniform vectors (X,Y)=(1,2),(3,6)(X,Y)=(1,2),(3,6) have mean (2,4)(2,4). Centered vectors are (−1,−2)(-1,-2) and (1,2)(1,2). Their squared coordinates average to 1 and 4; their product averages to 2.

Σ=(1224)\Sigma=\begin{pmatrix}1&2\\2&4\end{pmatrix}
Check your reasoning

Uniform feature vectors (X,Y)(X,Y): (0,1),(2,3)(0,1),(2,3). Give covariance rows.

  1. A(1,1);(1,1)(1,1);(1,1)
  2. B(2,3);(3,5)(2,3);(3,5)
  3. C(1,0);(0,1)(1,0);(0,1)
Show answer and explanation
(1,1);(1,1)(1,1);(1,1)

Variances (1,1)(1,1); covariance 11.

04PCA uses this joint variation to find directions of spread#

PCA uses this joint variation to find directions of spread. A covariance matrix is not a table of causal effects, and its numerical size depends on feature units. Changing a unit rescales both that feature’s variance and its covariances with others.

Key takeaway

Put variances on the diagonal; mirror each pair covariance.

  • Assemble the covariance matrix of a finite random vector.

Further questions

Must every covariance-matrix entry be positive?
No. Variances on the diagonal are nonnegative; off-diagonal covariances may be negative, zero or positive.

Sources & further reading

  1. [1]
  2. [2]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Covariance matrices.” Math behind ML. /learn/p-covariance-matrix