Expected values

01Weight by how often a value occurs#

An expectation is the average specified by a probability model. A possible value contributes in proportion to its probability. The model supplies the weights; the number of distinct outcomes does not.

Suppose $X$ is $0$ with probability $0.75$ and $4$ with probability $0.25$. In an imagined collection that follows those proportions exactly, three quarters contribute zero and one quarter contribute four. The average contribution is $1$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

Multiply each value by its probability, then add:

The index runs over the possible values. Every probability is nonnegative, and the complete set sums to one.

02Keep values and weights separate#

For values $-2,1,5$ with probabilities $0.5,0.3,0.2$, the contributions are $-1,0.3,1$. Their sum is $0.3$. A negative value contributes negatively; its probability remains positive.

Averaging the three values equally would answer a different question. Equal weighting is justified only when their probabilities are equal. Repeated observations can be summarized by frequencies, provided those frequencies supply the weights.

03A mean need not be an outcome#

When only $0$ and $4$ can occur, an expectation of $1$ does not predict that the next observation will be $1$. It describes the model's weighted average. Keep the value even when it is fractional or absent from the set of outcomes.

If $X$ represents a model's loss on a randomly drawn example, $E[X]$ is its expected loss. For losses $1$ and $9$ occurring with probabilities $0.9$ and $0.1$, expected loss is $1.8$. The rare large loss contributes $0.9$, as much as all the common small losses together.

Moving probability $\delta$ from a lower value $a$ to a higher value $b$ increases the mean by $\delta(b-a)$. This explains why changing the population weights can change expected performance without changing the loss values themselves.