StatisticsConcept reference

Log-likelihood

For positive independent-sample factors, log likelihood is the sum of their natural logarithms.

2 min read1 figureSources & further reading

On this page 9 sections

Overview
Understand the idea
For L=ab, use log L=log a+log b
An iid Bernoulli sample (1,0,1) has L(p)=p^2(1-p)
Model training often adds a negative log-likelihood over examples
Key takeaway
Further questions
Sources & further reading
Concept connections

Figure 1Log likelihood converts a product to a sum while retaining repeated factors. The product ab² corresponds to log a plus two copies of log b. Blocks show factors, not magnitudes.

01Understand the idea#

Multiplying many small likelihood factors can make a number too small for a computer to represent accurately. Taking logarithms turns the product into a sum. Because the logarithm increases with its input, maximizing a positive likelihood and maximizing its log give the same candidate.

For positive factors, $L=ab^2$ has one $a$ factor and two $b$ factors. Its log is $\log a+2\log b$ : every factor is retained.

\begin{aligned}\log(ab^2)&=\log a\\&\quad+2\log b\end{aligned}

Natural log turns an independent-sample product into a sum. Each factor must be positive for its log to be finite.

\begin{gathered}L(\theta)=\prod_i p_\theta(x_i)\\\ell(\theta)=\sum_i\log p_\theta(x_i)\end{gathered}

For likelihood factors $(1/2,1/4,1/2)$ , their product is $1/16$ . In base 2 the log-likelihood is $-1-2-1=-4$ , which is also $\log_2(1/16)$ . A negative log-likelihood changes the sign so that minimizing it corresponds to maximizing the original likelihood.

Check your reasoning

Positive factors $a,b,c$ . $L=a^3b$ . Give $\ell=\log L$ .

A $2\log a+\log b$
B $3\log a+\log b$
C $4\log a+\log b$

Show answer and explanation

$3\log a+\log b$

Log-term counts $(a,b,c)=(3,1,0)$ .

02For L=ab, use log L=log a+log b#

For $L=ab$ , use $\log L=\log a+\log b$ . Multiplying the logs applies the wrong rule. If a factor appears three times, its log appears three times too.

Check your reasoning

Positive factors $a,b,c$ . $L=a^2c$ . Reported $\ell=2\log a\log c$ . Repair.

A $2\log a+\log c$
B $2\log a\log c$
C $2\log a+2\log c$

Show answer and explanation

$2\log a+\log c$

Log-term counts $(a,b,c)=(2,0,1)$ .

03An iid Bernoulli sample (1,0,1) has L(p)=p^2(1-p)#

An iid Bernoulli sample $(1,0,1)$ has $L(p)=p^2(1-p)$ . For $0<p<1$ , its log is $2\log p+\log(1-p)$ . Natural log is increasing, so it preserves which positive likelihood is largest.

Check your reasoning

Ordered iid Bernoulli sample $(0,1,0,0)$ , $P(1)=p$ , $0<p<1$ . Give $\ell(p)$ .

A $3\log p+\log(1-p)$
B $\log p+2\log(1-p)$
C $\log p+3\log(1-p)$

Show answer and explanation

$\log p+3\log(1-p)$

1 one, 3 zeros: use those log-term counts.

04Model training often adds a negative log-likelihood over examples#

Model training often adds a negative log-likelihood over examples. Keep the log base and sign clear when comparing reported losses. Logging a sum of probabilities is not a replacement for summing the logs of the independent likelihood factors.

Key takeaway

Take one log term per positive factor. Keep multiplicities and every parameter-dependent term.

Express an independent-sample likelihood as a sum of log terms.

Further questions

Does a log likelihood supply a prior over parameters?

No. It rewrites the likelihood of fixed data. A prior is additional information used in a posterior update.

Sources & further reading

[1]
Pishro-Nik, §8.2.3 Maximum Likelihood Estimation ↗Textbook · Book

01Understand the idea#

02For L=ab, use log L=log a+log b#

03An iid Bernoulli sample (1,0,1) has L(p)=p^2(1-p)#

04Model training often adds a negative log-likelihood over examples#

Take one log term per positive factor. Keep multiplicities and every parameter-dependent term.

Further questions

Sources & further reading

Reference this concept