Covariance

01Understand the idea#

Covariance measures whether two variables tend to deviate from their means in the same direction. Pair deviations from the same observation, multiply them, and average with the distribution’s weights. Positive products represent same-direction deviations; negative products represent opposite-direction deviations.

Covariance averages how two variables deviate from their means together. A convenient calculation uses three expectations from the joint distribution.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

For binary $X,Y$ with joint rows $(0.1,0.2);(0.3,0.4)$, the means are 0.7 and 0.6. Only $(1,1)$ contributes to $E[XY]=0.4$.

Suppose equally likely outcomes are $(X,Y)=(1,2)$ and $(3,6)$. Their means are 2 and 4. Deviation products are $(-1)(-2)=2$ and $(1)(2)=2$, so covariance is 2. Pairing an $X$ from one outcome with a $Y$ from another would describe a different joint distribution.

02The binary table (0.1,0.2);(0.3,0.4) gives 0.4-0.7(0.6)=-0.02#

The binary table $(0.1,0.2);(0.3,0.4)$ gives $0.4-0.7(0.6)=-0.02$. Returning 0.4 would omit centering. Positive covariance favors same-sign centered deviations; negative covariance favors opposite signs.

03Use the actual support values, including their signs#

Use the actual support values, including their signs. Compute both means and the product mean before subtracting; joint cells need not have equal weights.

Zero covariance does not prove independence. Let $X$ be uniform on $-1,0,1$ and $Y=X^2$. Then $E[X]=E[XY]=0$, so covariance is zero.

Yet $P(X=0,Y=0)=1/3$, while $P(X=0)P(Y=0)=1/9$. This cell fails the independence test.

04A closer look#

Feature covariance helps describe which measurements vary together and supplies entries of the covariance matrix used by PCA. Its units are the product of the two variables’ units. Zero covariance rules out linear co-movement as measured here, but does not generally imply independence.

Further questions