Skip to content
Glacius
StatisticsConcept reference

Data leakage

Leakage uses information that should be unavailable to fitting, selection or the prediction.

2 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. Understand the idea
  3. A feature must also exist when the prediction is made
  4. For each cross-validation fit, learn preprocessing on that fold’s training blocks
  5. For time-ordered predictions, splitting randomly can expose future information
  6. Key takeaway
  7. Sources & further reading
  8. Concept connections

01Understand the idea#

Data leakage occurs when information unavailable at genuine prediction time influences model fitting or selection. Evaluation then answers an easier problem than the one the model will face. Leakage can enter through preprocessing, duplicated records, future data, or repeated decisions based on a test set.

Data leakage lets information shape a prediction or its evaluation when that information should be unavailable. Preprocessing is part of the fitted procedure: even a mean learned without labels can use evaluation information.

Training values 2 and 6 give mean 4. Center a held-out value of 10 by subtracting that fixed mean: 10−4=610-4=6. Including 10 while fitting the mean would leak held-out information.

Training values 2 and 6 give mean 4. Subtract this fixed training mean from held-out value 10 to obtain 6. Arrows connect training to mean, mean to output, and held-out value to output. The held-out value does not influence the learned mean.Training values 2 and 6 give mean 4. Subtract this fixed training mean from held-out value 10 to obtain 6. Arrows connect training to mean, mean to output, and held-out value to output. The held-out value does not influence the learned mean.
Figure 1Training values 2 and 6 give mean 4. Subtract this fixed training mean from held-out value 10 to obtain 6. Arrows connect training to mean, mean to output, and held-out value to output. The held-out value does not influence the learned mean.
Link to this figure ↗Download SVGDownload PNG

Suppose you standardize features using means computed from all rows, then split into training and test sets. Test observations already influenced the means. Instead fit the means on training rows, store them, and apply those same means to the held-out rows without refitting.

Check your reasoning

Center by the training mean. Training: 0, 4. Held-out value: 8. Fitted mean / centered value?

  1. A4 / 4
  2. B2 / 6
  3. C8 / 0
Show answer and explanation
2 / 6

Mean 2; 8−2=6.

02A feature must also exist when the prediction is made#

A feature must also exist when the prediction is made. At time 4, a reading available at time 3 is usable; one first available at time 7 is not. Availability exactly at the cutoff is allowed here.

Check your reasoning

Predict at time 6. Feature availability: a at 2, b at 9, c at 5. Usable IDs?

  1. Aa,b
  2. Ba,b,c
  3. Ca,c
Show answer and explanation
a,c

a,c available by 6.

03For each cross-validation fit, learn preprocessing on that fold’s training blocks#

For each cross-validation fit, learn preprocessing on that fold’s training blocks. Apply those learned settings unchanged to its validation block. Fitting a separate scaler on validation data still uses information outside the training fit.

Suppose test block J chooses the best setting. J has now shaped the model. If independent block K is still unused, freeze the procedure and evaluate on K. Renaming J cannot make it untouched.

Check your reasoning

Test D chose settings. Independent E is unused. “D is untouched.” Repair final evaluation.

  1. AFreeze; report D.
  2. BSelect on E; report E.
  3. CFreeze; evaluate E.
Show answer and explanation
Freeze; evaluate E.

D shaped selection.

04For time-ordered predictions, splitting randomly can expose future information#

For time-ordered predictions, splitting randomly can expose future information. For multiple rows from the same person or source, splitting related rows across sets can also inflate performance. Choose a split that reflects what will truly be new when the model is used.

Key takeaway

Trace when each value becomes available and which data shape every fitted or selected step.

  • Fit preprocessing within each training fold.

Sources & further reading

  1. [1]
  2. [2]

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Data leakage.” Math behind ML. /learn/s-leakage