Variance

01Measure spread around the mean#

Two distributions can have the same mean and very different variability. Variance measures how far values lie from their own mean by averaging squared deviations using the model's probabilities.

If $1$ and $5$ are equally likely, their mean is $3$. The deviations are $-2$ and $2$. Both squared deviations are $4$, so their weighted average is also $4$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

The definition says exactly which operations to perform:

where $\mu=E[X]$. Find the center, subtract it from each value, square, weight, and add.

02Unequal probabilities still matter#

Let $X$ equal $0$ with probability $0.75$ and $4$ with probability $0.25$. Its mean is $1$. The squared deviations are $1$ and $9$, whose contributions are $0.75$ and $2.25$.

The rare value contributes more to the variance because its squared distance is much larger. Averaging $1$ and $9$ equally would discard the probability model. Subtracting the mean after squaring would also change the calculation.

03What zero and units mean#

Signed deviations average to zero: negative and positive deviations cancel. Squaring before averaging prevents that cancellation. A model concentrated entirely at one value has zero variance; every deviation is zero.

If every value increases by $10$, the mean increases by $10$ too. Each deviation from the new mean stays the same, so variance is unchanged. If every deviation doubles, its square becomes four times as large.

Variance has squared units. If values measure seconds, variance measures seconds squared. This lesson uses a known probability model; estimating a population variance from a finite sample is a different task with a different justification.

For ML, this calculation describes the variability of quantities such as a sampled loss or gradient component. Its mean alone cannot tell you how much individual observations fluctuate.