Skip to content
Glacius
StatisticsConcept reference

Discrete likelihoods

A discrete likelihood evaluates the fixed sample joint PMF as a function of the candidate parameter. Independent factors multiply.

2 min read1 figureSources & further reading
On this page 9 sections
  1. Overview
  2. Understand the idea
  3. For the ordered sample (1,0) at p=0.7, use 0.7×0.3=0.21
  4. For categorical data, use the observed label to select each factor from the candidate PMF
  5. These likelihoods do not need to sum to 1 across candidates
  6. Key takeaway
  7. Further questions
  8. Sources & further reading
  9. Concept connections

01Understand the idea#

Likelihood compares candidate parameter settings by how much probability they assign to the data that actually occurred. The data stay fixed while the candidate changes. This is the direction used when fitting a probabilistic model.

Keep the observed sequence fixed at (1,0,1)(1,0,1). Candidate p=0.4p=0.4 assigns factors 0.4,0.6,0.40.4,0.6,0.4. Independence makes their product 0.0960.096.

The fixed ordered Bernoulli sample is (1,0,1). Its likelihood is p²(1−p). Candidate p=.4 gives .096 and p=.7 gives .147. The horizontal coordinate is the model parameter p; curve heights compare this same sample under different candidates.The fixed ordered Bernoulli sample is (1,0,1). Its likelihood is p²(1−p). Candidate p=.4 gives .096 and p=.7 gives .147. The horizontal coordinate is the model parameter p; curve heights compare this same sample under different candidates.
Figure 1The fixed ordered Bernoulli sample is (1,0,1). Its likelihood is p²(1−p). Candidate p=.4 gives .096 and p=.7 gives .147. The horizontal coordinate is the model parameter p; curve heights compare this same sample under different candidates.
Link to this figure ↗Download SVGDownload PNG

Likelihood is the sample probability viewed as a function of the candidate parameter. For independent observations, multiply the probability of every observed value.

L(θ)=∏ipθ(xi)L(\theta)=\prod_i p_\theta(x_i)

For an ordered independent binary sample (1,0,1)(1,0,1) and candidate success probability 0.4, the factors are 0.4,1−0.4,0.40.4,1-0.4,0.4. Multiply to get 0.0960.096. Candidate 0.7 instead gives 0.7(0.3)(0.7)=0.1470.7(0.3)(0.7)=0.147. The same observations favor the second candidate by likelihood.

Check your reasoning

Ordered iid Bernoulli sample (1,1,0)(1,1,0), P(1)=0.3P(1)=0.3. Give its likelihood.

Show answer and explanation
0.063

0.3×0.3×0.7=63/10000.3\times0.3\times0.7=63/1000.

02For the ordered sample (1,0) at p=0.7, use 0.7×0.3=0.21#

For the ordered sample (1,0)(1,0) at p=0.7p=0.7, use 0.7×0.3=0.210.7\times0.3=0.21. Adding the factors would give 1, which is not the probability of both observations. An ordered sequence also needs no count-combination factor.

Check your reasoning

Ordered iid Bernoulli sample (0,1)(0,1), P(1)=0.8P(1)=0.8. Reported 11 by adding factors. Repair.

  1. A1
  2. B0.16
  3. C0.84
Show answer and explanation
0.16

0.2×0.8=0.160.2\times0.8=0.16.

03For categorical data, use the observed label to select each factor from the candidate PMF#

For categorical data, use the observed label to select each factor from the candidate PMF. Repeated labels contribute repeated factors. Compare candidates on the same complete sample; their likelihoods need not sum to 1.

Check your reasoning

Ordered iid (2,1)(2,1). PMFs on (0,1,2)(0,1,2): A (0.25,0.5,0.25)(0.25,0.5,0.25); B (0.5,0.3,0.2)(0.5,0.3,0.2). (LA,LB)(L_A,L_B)?

  1. A(0.75,0.5)
  2. B(0.25,0.2)
  3. C(1/8,0.06)
Show answer and explanation
(1/8,0.06)

A: 0.25×0.50.25\times0.5; B: 0.2×0.30.2\times0.3.

04These likelihoods do not need to sum to 1 across candidates#

These likelihoods do not need to sum to 1 across candidates. They are probabilities of the fixed sample under separate models, not posterior probabilities of the models. Adding a prior and normalizing is a later Bayesian step. The ordered sequence here also differs from the event “two successes in any order.”

Key takeaway

For fixed ordered independent data, multiply observation probabilities under each candidate. Likelihood is not posterior probability.

  • Evaluate candidate parameters using a fixed discrete sample's probability model.

Further questions

What changes when the observations are continuous?
Evaluate and multiply density values for the same fixed observations. The result is a density likelihood, not a positive probability of exact observed points.

Sources & further reading

  1. [1]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Discrete likelihoods.” Math behind ML. /learn/s-likelihood