Multiple comparisons

01Testing many candidate effects creates many chances for a false positive#

Testing many candidate effects creates many chances for a false positive. Even if each test has a modest false-positive rate, searching a large collection and reporting only its most favorable result changes the meaning of the evidence.

A family is the full set of tests whose errors you want to control together. Fix it before seeing results. A false rejection rejects a true null hypothesis. Familywise error rate (FWER) is the probability of at least one such error.

Let each of three false-rejection events have probability at most 0.02. Their union means at least one error. The union bound adds the three bounds, giving at most 0.06.

If a family contains 20 tests and the desired family-wise false-positive bound is 0.05, Bonferroni uses threshold $0.05/20=0.0025$ per test. A p-value 0.01 passes an unadjusted 0.05 cutoff but fails this family threshold. The number of tests includes the defined family, not only the results selected for reporting.

To target FWER at most $\alpha$ across $m$ tests, give each test a bound $\alpha/m$. Adding all $m$ contributions gives $\alpha$. This is the equal-allocation Bonferroni rule.

02A closer look#

With a valid p-value, rejecting when $p\leq t$ gives a false-rejection probability at most $t$ when the null is true. Bonferroni therefore compares each p-value with $\alpha/m$. Equality meets the cutoff. A rejection is a decision, not proof that the null is false.

03The bound needs no independence assumption#

The bound needs no independence assumption. If individual valid bounds differ, add them. For bounds 0.02 and 0.03, FWER is at most 0.05. Cap sums above 1 at 1. A bound need not equal the actual error probability.

04A closer look#

Feature screening, repeated model comparisons, and many subgroup analyses can create multiplicity. Define the family and correction method before interpreting the strongest result. Different methods control different error quantities, so do not treat every adjusted threshold as answering the same question.