Skip to content
Glacius
StatisticsConcept reference

Logistic log loss

Binary log loss is the negative natural log probability of the label that occurred.

2 min read1 figureSources & further reading
On this page 7 sections
  1. Overview
  2. A logistic prediction gives the probability p of label 1
  3. A closer look
  4. Classification accuracy discards confidence after thresholding
  5. Key takeaway
  6. Sources & further reading
  7. Concept connections

01A logistic prediction gives the probability p of label 1#

A logistic prediction gives the probability p of label 1. Log loss evaluates the probability assigned to the label that actually occurred.

For y=1y=1, loss is −ln⁡p-\ln p. For y=0y=0, it is −ln⁡(1−p)-\ln(1-p). These natural logarithms give nats.

If the observed label is 0 while p is 0.8, the model assigned probability 0.2 to the observed label. Its loss is about 1.6094 nats.

For observed label y=0, model probability p=P(y=1) assigns realized-label probability 1−p. Loss is −ln(1−p). At p=.8, realized probability is .2 and loss is about 1.6094 nats. The graph uses natural logarithms.For observed label y=0, model probability p=P(y=1) assigns realized-label probability 1−p. Loss is −ln(1−p). At p=.8, realized probability is .2 and loss is about 1.6094 nats. The graph uses natural logarithms.
Figure 1For observed label y=0, model probability p=P(y=1) assigns realized-label probability 1−p. Loss is −ln(1−p). At p=.8, realized probability is .2 and loss is about 1.6094 nats. The graph uses natural logarithms.
Link to this figure ↗Download SVGDownload PNG
Check your reasoning

Label y=0y=0, predicted p(y=1)=0.75p(y=1)=0.75. Given ln⁡p=−0.2877\ln p=-0.2877 and ln⁡(1−p)=−1.3863\ln(1-p)=-1.3863, find log loss in nats.

Show answer and explanation
1.3863

The observed-label probability is 0.250.25. Negate its natural log: 1.38631.3863 nats.

02A closer look#

For independent labeled observations, their likelihoods multiply and their negative log-likelihoods add. A mean loss divides that sum by the number of observations.

Read whether a task asks for a total or an average. Do not take the negative log of an average probability; that changes the objective.

Check your reasoning

Labels: (1,0)(1, 0). Class 1 probabilities: (0.7,0.4)(0.7, 0.4). Their observed-label log probabilities are (−0.3567,−0.5108)(-0.3567, -0.5108). Find mean log loss in nats.

Show answer and explanation
0.43375

Negate and average: 0.8675/2=0.43375 nats.

03Classification accuracy discards confidence after thresholding#

Classification accuracy discards confidence after thresholding. Two models with the same accuracy can assign different probabilities to the realized labels.

Compare their actual log losses. Greater confidence in a wrong label produces a larger penalty.

Check your reasoning

Observed-label ln probabilities: A=(−0.5,−0.2)(-0.5,-0.2); B=(−0.1,−0.5)(-0.1,-0.5). Lower total log loss?

  1. AA
  2. BB
  3. CEqual
Show answer and explanation
B

Totals: A=0.7; B=0.6.

Key takeaway

Choose p for label 1 or 1−p for label 0, negate its log, and distinguish total loss from mean loss.

  • Compute binary negative log-likelihood for logistic predictions.

Sources & further reading

  1. [1]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Logistic log loss.” Math behind ML. /learn/s-logistic-loss