Skip to content
Glacius
StatisticsConcept reference

P-values

A p-value is the null-model probability of a statistic at least as extreme as observed under a stated rule.

2 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. Understand the idea
  3. An absolute-tail rule selects either sign
  4. The tail can have probability 0 or 1 under a supplied model
  5. A closer look
  6. Key takeaway
  7. Sources & further reading
  8. Concept connections

01Understand the idea#

A p-value measures how often the null model would produce a test statistic at least as extreme as the observed one, using the test’s specified direction. It asks about possible data under a model. It does not give the probability that the null model is true.

A p-value is a probability calculated under a specified null model. Sum the probabilities of statistic values at least as extreme as the observation, using the stated tail rule. For discrete values, “at least” includes ties.

This null law assigns masses 0.2, 0.5, 0.3 to T=−2,0,2. Observed t=2 and the rule T≥t select only T=2. Its null probability is 0.3.

The complete null distribution has masses .2,.5,.3 at test statistic −2,0,2. Observed statistic 2 selects the inclusive upper tail T≥2, here just the gold mass .3. The p-value is .3; it is not the probability the null is true.The complete null distribution has masses .2,.5,.3 at test statistic −2,0,2. Observed statistic 2 selects the inclusive upper tail T≥2, here just the gold mass .3. The p-value is .3; it is not the probability the null is true.
Figure 1The complete null distribution has masses .2,.5,.3 at test statistic −2,0,2. Observed statistic 2 selects the inclusive upper tail T≥2, here just the gold mass .3. The p-value is .3; it is not the probability the null is true.
Link to this figure ↗Download SVGDownload PNG

For an upper-tail test, suppose a null model gives probabilities 0.7, 0.2, and 0.1 to statistics 0, 1, and 2. Observing 1 includes both 1 and the more extreme 2, so the p-value is 0.2+0.1=0.30.2+0.1=0.3. Using only the observed statistic’s mass would omit more extreme outcomes.

Check your reasoning

Null masses T: -2:0.2, -1:0.3, 1:0.1, 2:0.4. Observed t=-2. Tail |T|≥|t|. p-value?

Show answer and explanation
0.6

Included mass: 0.2 + 0.4 = 0.6.

02An absolute-tail rule selects either sign#

An absolute-tail rule selects either sign. With masses 0.1,0.2,0.4,0.3 at T=−3,−1,0,2, observed t=−2 selects −3 and 2. Their mass is 0.4. The tails are asymmetric, so doubling one tail would give the wrong answer.

A CDF gives F(t)=P(T≤t). For a continuous null model, there is no mass exactly at t. A lower-tail p-value is F(t); an upper-tail p-value is 1−F(t). For F(1)=0.9, the upper tail is 0.1.

Check your reasoning

Continuous null CDF: F(3)=0.96. Observed t=3. Tail T≥t. p-value?

Show answer and explanation
0.04

1−F(t) = 0.04.

03The tail can have probability 0 or 1 under a supplied model#

The tail can have probability 0 or 1 under a supplied model. Even a small p-value does not assign a probability to H0 being true. It describes how often the stated statistic tail occurs if that null model generates the data.

Check your reasoning

Null masses T: 0:0.6, 1:0.25, 2:0.15. Observed t=1. Tail T≥t. “Probability H0 is true.” Repair value / meaning.

  1. A0.4; H0 is true.
  2. B0.6; tail under H0.
  3. C0.4; tail under H0.
Show answer and explanation
0.4; tail under H0.

0.25+0.15=0.4.

04A closer look#

When comparing a model improvement with a no-improvement null, report the effect estimate and uncertainty along with the p-value. A small p-value need not mean the effect is practically large, and a large p-value does not establish exact equality or prove the null.

Key takeaway

Use the supplied null law and tail rule, include tied discrete values, and sum their probabilities.

  • Compute the stated null tail probability.

Sources & further reading

  1. [1]
  2. [2]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “P-values.” Math behind ML. /learn/s-pvalue