Skip to content
Glacius
StatisticsConcept reference

Cross-validation

Cross-validation evaluates each development fold with a model fitted on the other folds.

2 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. Cross-validation estimates how a training procedure performs on data it did not fit
  3. This lesson reports pooled loss per validated observation
  4. A closer look
  5. Any learned preprocessing belongs inside each training fold
  6. Key takeaway
  7. Sources & further reading
  8. Concept connections

01Cross-validation estimates how a training procedure performs on data it did not fit#

Cross-validation estimates how a training procedure performs on data it did not fit. Split the training data into folds, fit on the other folds, and evaluate on the held-out fold. Rotating this role lets each record receive an out-of-fold evaluation.

Cross-validation rotates which development observations are held out. For each fold, fit a new model on the other folds and evaluate on that fold. The held-out predictions come from models that did not fit those observations.

For validation fold B, fit on A and C. Repeat with A and C held out in turn. Keep final test T outside this procedure.

Three development blocks A,B,C rotate roles. Row 1 validates A and trains B,C; row 2 validates B and trains A,C; row 3 validates C and trains A,B. Final test T is reserved outside every row. The table expresses roles, not a metric or independent fold scores. A star marks the validation block in each row, so the roles remain explicit without color.Three development blocks A,B,C rotate roles. Row 1 validates A and trains B,C; row 2 validates B and trains A,C; row 3 validates C and trains A,B. Final test T is reserved outside every row. The table expresses roles, not a metric or independent fold scores. A star marks the validation block in each row, so the roles remain explicit without color.
Figure 1Three development blocks A,B,C rotate roles. Row 1 validates A and trains B,C; row 2 validates B and trains A,C; row 3 validates C and trains A,B. Final test T is reserved outside every row. The table expresses roles, not a metric or independent fold scores. A star marks the validation block in each row, so the roles remain explicit without color.
Link to this figure ↗Download SVGDownload PNG

With five folds, one run trains on four and evaluates on the remaining one. Repeat until each fold has been held out once. If fold sizes differ, an overall per-record loss weights each fold mean by its evaluation count; a simple average of fold means answers a different weighting question.

Check your reasoning

Development folds X={a,b}, Y={c,d}, Z={e,f}. Final test T. Validate X. Fit / validate blocks?

  1. AY,Z / X.
  2. BX,Y,Z / X.
  3. CY,Z,T / X.
Show answer and explanation
Y,Z / X.

Fit excludes X and T.

02This lesson reports pooled loss per validated observation#

This lesson reports pooled loss per validated observation. Add the validation loss totals, then divide by the number of validated observations. With fold counts 2 and 4 and loss totals 6 and 6, the pooled mean is 12/6=212/6=2.

Unequal fold sizes need their observation counts. The fold means above are 3 and 1.5; their unweighted mean 2.25 differs from the pooled mean 2. With equal fold sizes, the two averaging rules coincide.

Check your reasoning

Validation folds (n, loss total): (1,2), (3,6). Each fold excluded from its fit. Pooled mean per observation?

Show answer and explanation
2

Total 8 / count 4 = 2.

03A closer look#

Do not replace the aggregate with a favorable fold or use the final test to select settings. For time-dependent data, validation should reflect future observations; group-dependent data need suitable group separation. Random folds are not universal.

Check your reasoning

Validation folds (n, loss total): (2,4), (2,8). “CV=2: best fold.” Repair pooled mean per observation.

  1. A2
  2. B12
  3. C3
Show answer and explanation
3

Pool: 12/4=3.

04Any learned preprocessing belongs inside each training fold#

Any learned preprocessing belongs inside each training fold. Reusing a transformation fitted on every row gives the held-out fold influence over training. After selecting a procedure, a separate untouched test set can assess the full selection process.

Key takeaway

Follow the supplied split roles, pool the requested validation losses using their counts, and preserve an untouched final test.

  • Aggregate held-out validation losses under the stated fold scheme.

Sources & further reading

  1. [1]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Cross-validation.” Math behind ML. /learn/s-cross-validation