Empirical objectives

01A loss for the whole dataset#

An empirical objective summarizes how a model performs on a fixed dataset. First calculate a loss for every example using the same rule. Then average those losses. Changing the parameters changes the losses and therefore the objective.

At one parameter setting, three examples have losses $1,4,7$. Their total is $12$, and the dataset contains three examples. The empirical average is $4$.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

Writing the parameter vector as $\theta$ and example $i$'s loss as $\ell_i(\theta)$ gives

Each example contributes once, even if several losses have the same value.

02Loss first, average second#

Suppose predictions are $(0,3)$ and targets are $(2,1)$. The residuals are $-2$ and $2$. Under half-squared loss, each example contributes $2$, giving mean loss $2$.

Averaging residuals first would give zero, then squaring would give zero. That would hide both errors. The order of operations is part of the objective's definition.

03State the convention#

Squared loss $(\hat y-y)^2$ and half-squared loss $(\hat y-y)^2/2$ differ by a factor of two. Summed loss and mean loss differ by the number of examples. Read the stated convention before calculating.

Repeating the entire dataset leaves its mean loss unchanged: both numerator and denominator double. Repeating only its hardest example changes that example's weight and usually changes the mean.

A zero-loss example still belongs in the denominator. Averaging batch averages equally works only when the batches have equal size; otherwise weight each average by its number of examples.

This objective describes the observed dataset. Expected loss describes a probability model for new examples. Treating the empirical average as an estimate of that expectation requires assumptions about how the data were collected. Model evaluation checks that connection separately.