Entropy

01Entropy is the average surprise of outcomes drawn from a distribution#

Entropy is the average surprise of outcomes drawn from a distribution. It summarizes uncertainty before an outcome is observed. A distribution concentrated on one certain outcome has no uncertainty; a spread across several plausible outcomes can have more.

Entropy is the expected surprisal of a finite distribution. Find each outcome’s surprisal, then average using its own probability.

For masses 0.5,0.125,0.125,0.125,0.125, the first contribution is 0.5 bits. Each small outcome adds 0.375 bits, giving entropy 2 bits.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

For probabilities $(1/2,1/4,1/4)$, the surprises are $(1,2,2)$ bits. Weight each by how often that outcome occurs: $0.5(1)+0.25(2)+0.25(2)=1.5$ bits. Averaging the three surprises equally would incorrectly treat the categories as equally likely.

02Category counts define an empirical distribution after you divide by their total#

Category counts define an empirical distribution after you divide by their total. Use those probabilities as weights, rather than assuming the categories are equally likely.

Entropy does not change when you rename the categories. It depends on how the probability is distributed.

03A zero-probability outcome contributes 0 by the convention 0log 0=0#

A zero-probability outcome contributes 0 by the convention $0\log 0=0$. A distribution concentrated on one outcome has entropy 0.

Zero entropy means certainty under that distribution. A forecaster can be certain and wrong, so entropy alone cannot establish factual correctness.

04A closer look#

Entropy supplies a baseline expected log loss when a forecast matches the target distribution. It describes the stated distribution, not how confident a learner feels. A zero-probability category contributes zero to the expectation; it is never drawn from that distribution.