Skip to content
Glacius
OptimizationConcept reference

Empirical objectives

An empirical objective is the average stated loss over all examples in a fixed dataset.

1 min read1 figureSources & further reading
On this page 7 sections
  1. Overview
  2. An empirical objective averages per-example losses over a fixed dataset
  3. If predictions and targets are supplied, compute each loss before averaging
  4. The denominator is the number of examples, not the number of distinct loss values
  5. Key takeaway
  6. Sources & further reading
  7. Concept connections

01An empirical objective averages per-example losses over a fixed dataset#

An empirical objective averages per-example losses over a fixed dataset. Every example counts, including repeated values and zero losses.

J(θ)=1n∑i=1nℓi(θ)J(\theta)=\frac1n\sum_{i=1}^n\ell_i(\theta)

Three examples give losses 1, 4 and 7. Sum all three, then divide by their count.

Three fixed examples have losses 1,4,7. Their sum is 12; their empirical mean is 4. The last bar shows the average on the same loss scale.Three fixed examples have losses 1,4,7. Their sum is 12; their empirical mean is 4. The last bar shows the average on the same loss scale.
Figure 1Three fixed examples have losses 1,4,7. Their sum is 12; their empirical mean is 4. The last bar shows the average on the same loss scale.
Link to this figure ↗Download SVGDownload PNG
Check your reasoning

Losses: 0,3,3,60, 3, 3, 6. Find their empirical average.

Show answer and explanation
3

Sum 1212 over 44 examples: 33.

02If predictions and targets are supplied, compute each loss before averaging#

If predictions and targets are supplied, compute each loss before averaging. Squaring the mean residual can give a different result.

Follow the stated convention: squared loss and half-squared loss differ by a factor of two.

Check your reasoning

Targets: (1,0)(1, 0); predictions: (4,2)(4, 2). Each loss is (y^−y)2/2(\hat y-y)^2/2. Find the mean loss.

Show answer and explanation
3.25

Per-example losses: 4.5, 2. Their average is 3.253.25.

03The denominator is the number of examples, not the number of distinct loss values#

The denominator is the number of examples, not the number of distinct loss values. A zero-loss example still belongs in that count.

This average describes the observed dataset. It need not equal the unknown expected loss on new population draws.

Check your reasoning

Losses: 0,0,4,4,20, 0, 4, 4, 2. A draft divides the sum by 3 distinct losses. Correct the average.

Show answer and explanation
2

Sum 1010 over 55 examples: 22.

Key takeaway

Compute every loss with its stated convention, sum them and divide by the actual example count.

  • Form an average per-example loss over a fixed dataset.

Sources & further reading

  1. [1]
    Dive into Deep Learning §12.5.2 ↗Dive into Deep Learning §12.5.2 · Article

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Empirical objectives.” Math behind ML. /learn/o-empirical