Skip to content
Glacius
ProbabilityConcept reference

Cross-entropy

Cross-entropy is the target-weighted average of the forecast’s negative log probabilities.

2 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. Cross-entropy evaluates a forecast using outcomes from a target distribution
  3. For observed category counts, divide by the total to obtain target frequencies
  4. If Q assigns probability 0 to any outcome with positive P mass, cross-entropy is infinite
  5. For observed labels, target frequencies are counts divided by the number of examples
  6. Key takeaway
  7. Sources & further reading
  8. Concept connections

01Cross-entropy evaluates a forecast using outcomes from a target distribution#

Cross-entropy evaluates a forecast using outcomes from a target distribution. The target tells you how often each category occurs; the forecast tells you how surprising that category would be. Separating those two roles is essential for understanding classification log loss.

Cross-entropy measures how much surprise a forecast assigns to outcomes drawn from a target distribution. The target P supplies the weights; forecast Q supplies the probabilities inside the log.

H(P,Q)=−∑ipilog⁡2qi\begin{gathered}H(P,Q)=\\-\sum_i p_i\log_2 q_i\end{gathered}

Use target P=(0.625,0.25,0.125) and forecast Q=(0.5,0.25,0.25). Forecast surprisals are 1,2,2 bits; the weighted contributions total 1.375 bits.

cross-entropy: category width is target probability P; rectangle height is −log₂ Q in bits. P=(0.625, 0.25, 0.125); Q=(0.5, 0.25, 0.25).cross-entropy: category width is target probability P; rectangle height is −log₂ Q in bits. P=(0.625, 0.25, 0.125); Q=(0.5, 0.25, 0.25).
Figure 1cross-entropy: category width is target probability P; rectangle height is −log₂ Q in bits. P=(0.625, 0.25, 0.125); Q=(0.5, 0.25, 0.25).
Link to this figure ↗Download SVGDownload PNG

Suppose the target is P=(0.75,0.25)P=(0.75,0.25) and the forecast is Q=(0.5,0.5)Q=(0.5,0.5). The forecast assigns 1 bit of surprise to either outcome. Weight by the target frequencies: 0.75(1)+0.25(1)=10.75(1)+0.25(1)=1 bit. Replacing the target weights with forecast weights would evaluate a different expectation.

Check your reasoning

In the same category order, target P: (0.5,0.25,0.25)(0.5, 0.25, 0.25); forecast Q: (0.25,0.5,0.25)(0.25, 0.5, 0.25). Find cross-entropy H(P,Q) in bits.

Show answer and explanation
1.75

P-weighted forecast surprisals: 1, 0.25, 0.5; total: 1.751.75 bits.

02For observed category counts, divide by the total to obtain target frequencies#

For observed category counts, divide by the total to obtain target frequencies. Averaging forecast log loss over observations gives this empirical cross-entropy.

Weighting with Q instead would evaluate a different quantity. Match categories before multiplying their target weights and forecast surprisals.

Check your reasoning

Target category counts: (2,1,1)(2, 1, 1). Forecast Q: (0.25,0.25,0.5)(0.25, 0.25, 0.5). In the same category order, find mean log loss in bits.

Show answer and explanation
1.75

P-weighted forecast surprisals: 1, 0.5, 0.25; total: 1.751.75 bits.

03If Q assigns probability 0 to any outcome with positive P mass, cross-entropy is infinite#

If Q assigns probability 0 to any outcome with positive P mass, cross-entropy is infinite. That outcome cannot simply be dropped.

An outcome with P mass 0 contributes 0, including when Q is also 0 there. Inspect the target mass before deciding what a zero forecast means.

Check your reasoning

P=Q=(0,0.5,0.5)(0, 0.5, 0.5). Draft: zero Q makes loss infinite. Correct H(P,Q), in bits?

  1. AInfinite
  2. B1
  3. C2
Show answer and explanation
1

Zero-target term: 0. Others: 0.5+0.5=1 bit.

04For observed labels, target frequencies are counts divided by the number of examples#

For observed labels, target frequencies are counts divided by the number of examples. Averaging the forecast’s log loss over those examples gives empirical cross-entropy. A forecast probability zero for an outcome that occurs makes its log loss infinite, rather than removing that outcome from the average.

Key takeaway

Use target weights and forecast logs; positive target mass at a zero forecast makes the result infinite.

  • Compute expected negative log probability under a supplied target distribution.

Sources & further reading

  1. [1]
  2. [2]
    Dive into Deep Learning §22.11 ↗Dive into Deep Learning §22.11 · Article

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Cross-entropy.” Math behind ML. /learn/p-cross-entropy