ProbabilityConcept reference

KL divergence

KL(P∥Q) is target-weighted log probability ratio, equal to cross-entropy minus target entropy.

2 min read2 figuresSources & further reading

On this page 7 sections

Overview
KL as excess log loss
Computing the divergence
Zero probabilities and argument order
Key takeaway
Sources & further reading
Concept connections

Figure 1kl: category width is target probability P; rectangle height is −log₂ Q in bits. Dashed tops show −log₂ P. KL is the total signed area difference, not the unsigned space between shapes. P=(0.75, 0.25); Q=(0.3, 0.7).

01KL as excess log loss#

KL divergence measures the extra expected log loss caused by using a forecast $Q$ when outcomes follow $P$ . The target’s own entropy is unavoidable uncertainty; cross-entropy includes that baseline plus the cost of mismatch. Their difference isolates the mismatch.

KL divergence measures excess expected log loss from using forecast Q when the target distribution is P, with that argument order.

\begin{gathered}D(P\Vert Q)\\=H(P,Q)-H(P)\end{gathered}

Target entropy 0.75 bits and cross-entropy 1.25 bits give KL 0.5 bits. Subtract the target’s own uncertainty to isolate the excess loss.

Figure 2H(P)=0.75 bits. H(P,Q)=1.25 bits. Both bars use the same scale. The excess is 0.5 bits.

If target entropy is 0.75 bits and forecast cross-entropy is 1.25 bits, the excess is $1.25-0.75=0.5$ bits. The order $D(P\Vert Q)$ matters: $P$ supplies the expectation weights. Reversing the distributions asks a different question and can even change a finite value to infinity.

Check your reasoning

$H(P)=1.5$ bits and $H(P,Q)=1.75$ bits. Find $D(P\Vert Q)$ in bits.

Show answer and explanation

0.25

Subtract target entropy from cross-entropy: $1.75-1.5=0.25$ bits.

02Computing the divergence#

You can also compute directly: use P as weights on the log probability ratios.

\begin{gathered}D(P\Vert Q)=\\\sum_i p_i\log_2\frac{p_i}{q_i}\end{gathered}

Individual log ratios may be negative; the complete divergence is nonnegative.

Check your reasoning

Same order: P= $(0.25, 0.25, 0.5)$ , Q= $(0.25, 0.5, 0.25)$ . Use log₂(P/Q)= $(0, -1, 1)$ . Find D(P∥Q) in bits, rounded to 4 decimals.

Show answer and explanation

0.25

P-weighted terms: 0−0.25+0.5=0.25 bits.

03Zero probabilities and argument order#

Zero P mass contributes zero. Positive P mass where Q is zero makes D(P∥Q) infinite.

KL need not stay the same when its arguments are swapped. It is not a symmetric distance, even though some particular pairs have equal values.

Check your reasoning

Same order: P= $(0, 1)$ , Q= $(0.75, 0.25)$ . Draft: KL is symmetric. Correct (D(P∥Q), D(Q∥P)), bits.

A(∞, 2)
B(2, ∞)
C(0, 0)

Show answer and explanation

(2, ∞)

Reverse support fails.

Comparing probability models with KL requires specifying which distribution supplies the outcomes. Although it is nonnegative, it is not a distance metric: it need not be symmetric. Individual weighted log-ratio terms can be negative, so sum every category before interpreting the total.

Key takeaway

Keep P as the weights, preserve argument order, and check support before calculating.

Compute a finite-distribution KL divergence with valid support handling.

Sources & further reading

[1]
Cosma Shalizi: Information Theory I, KL and support conventions ↗Cosma Shalizi · Article
[2]
Dive into Deep Learning §22.11 ↗Dive into Deep Learning §22.11 · Article

01KL as excess log loss#

02Computing the divergence#

03Zero probabilities and argument order#

Keep P as the weights, preserve argument order, and check support before calculating.

Sources & further reading

Reference this concept