Skip to content
Glacius
ProbabilityConcept reference

Surprisal

The surprisal of a positive-probability event is the negative logarithm of its probability.

2 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. Surprisal measures how unexpected an outcome is under a probability forecast
  3. When an event contains several equally likely outcomes, find its total probability first
  4. A certain event has probability 1 and surprisal 0 because log_2 1=0
  5. A closer look
  6. Key takeaway
  7. Sources & further reading
  8. Concept connections

01Surprisal measures how unexpected an outcome is under a probability forecast#

Surprisal measures how unexpected an outcome is under a probability forecast. A high-probability outcome carries little surprise; an outcome assigned a tiny probability carries much more. Using a logarithm makes surprises add when independent event probabilities multiply.

Surprisal assigns more information to a less likely event. For a positive event probability p, measured in bits:

I=−log⁡2pI=-\log_2 p

An event with probability 1/128=2−71/128=2^{-7} has surprisal 7 bits. The logarithm reads its exponent; the minus sign makes the result positive.

The curve uses event probability p on a linear horizontal scale and information in bits vertically. At p=1/128, surprisal is 7 bits. At p=1 it is 0. Zero probability has infinite surprisal and no finite plotted point.The curve uses event probability p on a linear horizontal scale and information in bits vertically. At p=1/128, surprisal is 7 bits. At p=1 it is 0. Zero probability has infinite surprisal and no finite plotted point.
Figure 1The curve uses event probability p on a linear horizontal scale and information in bits vertically. At p=1/128, surprisal is 7 bits. At p=1 it is 0. Zero probability has infinite surprisal and no finite plotted point.
Link to this figure ↗Download SVGDownload PNG

In bits, probability 1/21/2 gives −log⁡2(1/2)=1-\log_2(1/2)=1, and probability 1/81/8 gives 3. Two independent half-probability outcomes have joint probability 1/41/4 and total surprise 2 bits. The minus sign makes these values nonnegative because logarithms of probabilities below 1 are negative.

Check your reasoning

An event has probability 0.250.25. Find its surprisal in bits.

Show answer and explanation
2

−log⁡2(0.25)=2-\log_2(0.25)=2 bits. The negative sign makes this nonnegative.

02When an event contains several equally likely outcomes, find its total probability first#

When an event contains several equally likely outcomes, find its total probability first. Counting outcomes alone is not the information value.

The log base fixes the unit. Base 2 gives bits; a natural logarithm would give nats.

Check your reasoning

Of 3232 equally likely labeled tickets, 44 are marked. Find the marked event’s surprisal in bits.

Show answer and explanation
3

−log⁡2(0.125)=3-\log_2(0.125)=3 bits. The negative sign makes this nonnegative.

03A certain event has probability 1 and surprisal 0 because log_2 1=0#

A certain event has probability 1 and surprisal 0 because log⁡21=0\log_2 1=0. Positive probabilities below 1 have positive surprisal.

This measures surprise relative to a probability model. A surprising observation does not by itself prove a model is wrong.

Check your reasoning

Probability is 0.0156250.015625. A draft uses log⁡2p\log_2 p as surprisal. Give the corrected bit value.

Show answer and explanation
6

−log⁡2(0.015625)=6-\log_2(0.015625)=6 bits. The negative sign makes this nonnegative.

04A closer look#

A probabilistic model’s log loss uses this same idea: confidently assigning tiny probability to the observed outcome is costly. State the logarithm base, since base 2 gives bits while natural logarithms give nats. The numerical units change, though forecast ranking does not.

Key takeaway

Use −log₂p for bits: rarer events carry more surprisal; a certain event carries zero.

  • Compute the information associated with a positive-probability event.

Sources & further reading

  1. [1]
    Dive into Deep Learning §22.11.1.1 ↗Dive into Deep Learning §22.11.1.1 · Article

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Surprisal.” Math behind ML. /learn/p-surprisal