Skip to content
Glacius
StatisticsConcept reference

Holdout evaluation

A holdout set stays outside fitting and selection until the model is frozen.

2 min read1 figureSources & further reading
On this page 7 sections
  1. Overview
  2. A holdout set evaluates a model on observations that did not shape it
  3. Choosing settings also shapes the model
  4. A closer look
  5. Key takeaway
  6. Sources & further reading
  7. Concept connections

01A holdout set evaluates a model on observations that did not shape it#

A holdout set evaluates a model on observations that did not shape it. Use training data to fit the model. Freeze the complete procedure before reading the final holdout results. Here the observations are independent draws from the target distribution.

Train on a, b, c, and reserve d, e. After freezing the model, suppose the two holdout losses are 1 and 5. Its holdout mean loss is (1+5)/2=3(1+5)/2=3.

Training records a,b,c are used to fit the model. The model is frozen before it and untouched holdout records d,e produce the score. No held-out information flows back into the model fit. Boxes indicate roles, not relative sample sizes.Training records a,b,c are used to fit the model. The model is frozen before it and untouched holdout records d,e produce the score. No held-out information flows back into the model fit. Boxes indicate roles, not relative sample sizes.
Figure 1Training records a,b,c are used to fit the model. The model is frozen before it and untouched holdout records d,e produce the score. No held-out information flows back into the model fit. Boxes indicate roles, not relative sample sizes.
Link to this figure ↗Download SVGDownload PNG
Check your reasoning

Frozen model. IID records (T=fit, H=untouched): a:T:1, b:H:2, c:H:6. Values are losses. Holdout mean?

Show answer and explanation
4

Holdout loss: 2+6; divide by 2.

02Choosing settings also shapes the model#

Choosing settings also shapes the model. A validation block can guide those choices, but a separate test block must remain untouched for final evaluation. Calling a reused block “test” does not restore independence.

Check your reasoning

Independent IID blocks A,B,C. Fit on B; choose settings on C. Freeze. Untouched final evaluation block(s)?

  1. AB
  2. BA
  3. CC
Show answer and explanation
A

A is still unused.

03A closer look#

A model can fit training data unusually well, so training loss alone is not a fresh evaluation. A holdout mean is still a sample result, not an exact future error. Its relevance also depends on evaluating the population the model will face.

Check your reasoning

Independent IID blocks K,L,M. Fit on L; choose settings on K. Freeze. “K is untouched.” Final block?

  1. AL
  2. BK
  3. CM
Show answer and explanation
M

M is still unused.

Key takeaway

Trace every use of the data. Freeze fitting and selection, then average losses on the untouched target-distribution observations.

  • Separate fitting and selection from final evaluation.

Sources & further reading

  1. [1]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Holdout evaluation.” Math behind ML. /learn/s-holdout