Skip to content
Glacius
StatisticsConcept reference

Bootstrap resampling

An IID nonparametric bootstrap repeatedly draws with replacement from the empirical sample at its original size and recomputes the statistic.

2 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. Understand the idea
  3. Generate another same-size replacement sample and evaluate the same rule again
  4. Distinct records can share a value
  5. A closer look
  6. Key takeaway
  7. Sources & further reading
  8. Concept connections

01Understand the idea#

The bootstrap estimates how a statistic varies across samples by resampling the observations you already have. It treats their empirical distribution as an approximation to the population. Each resample has the original size and is drawn with replacement, so records may repeat or be omitted.

For an IID sample, the ordinary bootstrap approximates sampling variation by resampling observed records. Draw independently and uniformly with replacement, using the original sample size. Apply the same statistic to each resample.

For records a=2,b=5,c=8a=2,b=5,c=8, one possible draw sequence is a,a,ca,a,c. It produces 2,2,82,2,8: three draws, with a repeat and an omitted record.

The original IID sample has three records: a=2,b=5,c=8. One possible uniform with-replacement sequence is a,a,c, producing the same-size resample 2,2,8. Two arrows from original a show that it is drawn twice; original b is omitted. The resampled mean is (2+2+8)/3=4.The original IID sample has three records: a=2,b=5,c=8. One possible uniform with-replacement sequence is a,a,c, producing the same-size resample 2,2,8. Two arrows from original a show that it is drawn twice; original b is omitted. The resampled mean is (2+2+8)/3=4.
Figure 1The original IID sample has three records: a=2,b=5,c=8. One possible uniform with-replacement sequence is a,a,c, producing the same-size resample 2,2,8. Two arrows from original a show that it is drawn twice; original b is omitted. The resampled mean is (2+2+8)/3=4.
Link to this figure ↗Download SVGDownload PNG

From records a=2,b=5,c=8a=2,b=5,c=8, a draw sequence (a,a,c)(a,a,c) gives values (2,2,8)(2,2,8) and mean 4. A different sequence (b,c,c)(b,c,c) gives (5,8,8)(5,8,8) and mean 7. Compute the statistic separately for every resample; do not report the unchanged original mean 5 for each one.

For the supplied mean rule, add and divide by 33. The resample 2,2,82,2,8 gives 12/3=412/3=4. The original mean is 55; the bootstrap replicate is recomputed from its own draws.

Check your reasoning

Original IID records: a=1, b=4, c=10. Draw IDs b,b,c with replacement. T=sum/3. Resample and T?

  1. A(4, 4, 10); T=6.
  2. B(1, 4, 10); T=5.
  3. C(4, 4, 10); T=5.
Show answer and explanation
(4, 4, 10); T=6.

Mean: 18/3=6.

02Generate another same-size replacement sample and evaluate the same rule again#

Generate another same-size replacement sample and evaluate the same rule again. The collected statistic values approximate its sampling distribution. Keep one output per resample, including duplicate outputs.

Check your reasoning

Original IID records: a=0, b=10. Bootstrap ID runs: (a,a), (b,b), (b,a). T=(x+y)/2. Outputs?

  1. A(5, 5, 5)
  2. B(0, 10, 5)
  3. C(0, 20, 10)
Show answer and explanation
(0, 10, 5)

Means: 0, 10, 5.

03Distinct records can share a value#

Distinct records can share a value. For observed 2,2,82,2,8, a draw has value 22 with probability 2/32/3. Deduplicating would change the empirical distribution. Tiny examples show the mechanics; they do not guarantee a good population approximation.

Check your reasoning

Original IID sample has 2 records. “Discard repeated bootstrap draws.” Repair.

  1. ARedraw repeated IDs.
  2. BKeep unique IDs only.
  3. CKeep all 2 draws.
Show answer and explanation
Keep all 2 draws.

Replacement allows repeats.

04A closer look#

The spread of bootstrap means approximates sampling uncertainty when resampling matches the data-generating process. It creates no new real-world observations. Time series and grouped data may need a different design that respects dependence, instead of ordinary IID resampling.

Key takeaway

Draw observed records independently with replacement, keep the original sample size, and recompute the same statistic once per resample.

  • Construct same-size replacement resamples.

Sources & further reading

  1. [1]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Bootstrap resampling.” Math behind ML. /learn/s-bootstrap