Skip to content
Glacius
OptimizationConcept reference

Objective functions

A scalar objective maps the specified decision variables to one numerical cost.

2 min read1 figureSources & further reading
On this page 8 sections
  1. Overview
  2. An objective turns a model-design goal into a scalar score
  3. The same construction works for resource costs
  4. In a prediction error, the observed target is fixed data
  5. A training objective is a chosen proxy for what you care about
  6. Key takeaway
  7. Sources & further reading
  8. Concept connections

01An objective turns a model-design goal into a scalar score#

An objective turns a model-design goal into a scalar score. Training compares parameter choices by this score and tries to make it smaller. You must decide what counts as an error and how errors are combined before an optimization method can help.

An objective assigns one numerical score to each decision. To minimize it, first state what you can adjust and how its costs combine.

Fixed data and cost coefficients stay fixed while decisions vary.

Adjust x and y. A squared x cost weighted by 7 and a squared y cost weighted by 4 give one total:

Objective f(x,y)=7x²+4y² assigns a scalar cost to each decision pair. Three ellipses show levels 4,8,12; the center (0,0) has cost 0. Coefficients stay fixed. Equal coordinate units preserve the elliptical geometry.Objective f(x,y)=7x²+4y² assigns a scalar cost to each decision pair. Three ellipses show levels 4,8,12; the center (0,0) has cost 0. Coefficients stay fixed. Equal coordinate units preserve the elliptical geometry.
Figure 1Objective f(x,y)=7x²+4y² assigns a scalar cost to each decision pair. Three ellipses show levels 4,8,12; the center (0,0) has cost 0. Coefficients stay fixed. Equal coordinate units preserve the elliptical geometry.
Link to this figure ↗Download SVGDownload PNG

If two examples have losses 2 and 6, the sum objective is 8 while the mean objective is 4. They rank these parameter settings in the same order when the dataset is fixed, but their gradients have different scales. Read whether a task asks for a sum, mean, or additional penalty.

Check your reasoning

Adjust x,yx,y. The costs are 55 times x2x^2 and 99 times y2y^2. Choose their total objective.

  1. A9x2+5y29x^2+5y^2
  2. B5x2+9y25x^2+9y^2
  3. C5x2−9y25x^2-9y^2
Show answer and explanation
5x2+9y25x^2+9y^2

Add the weighted costs.

02The same construction works for resource costs#

The same construction works for resource costs. Multiply each adjustable quantity by its stated unit cost, then add the contributions.

An objective describes how to score decisions. It does not yet tell you which decisions are allowed or how to find the best one.

Check your reasoning

Adjust job counts x,yx,y. Unit costs are fixed at 44 and 99, respectively. Choose total cost.

  1. A9x+4y9x+4y
  2. B4x4x
  3. C4x+9y4x+9y
Show answer and explanation
4x+9y4x+9y

Sum cost × count.

03In a prediction error, the observed target is fixed data#

In a prediction error, the observed target is fixed data. Changing that target to make the score smaller changes the problem.

List the decision variables explicitly. A symbol appearing in a formula is not automatically adjustable.

Check your reasoning

Target t=7t=7 is fixed. For loss (x−t)2(x-t)^2, a draft adjusts x and t. Repair it.

  1. AAdjust x: (x−7)2(x-7)^2
  2. BAdjust both: (x−t)2(x-t)^2
  3. CAdjust x: x2−7x^2-7
Show answer and explanation
Adjust x: (x−7)2(x-7)^2

Only x varies; t stays 7.

04A training objective is a chosen proxy for what you care about#

A training objective is a chosen proxy for what you care about. Minimizing squared prediction error does not automatically guarantee calibrated probabilities, fairness, or good performance on new data. Those require their own definitions and evaluation.

Key takeaway

Name the adjustable decisions, hold data fixed and combine the stated costs into one scalar.

  • Express a scalar objective for specified decision variables.

Sources & further reading

  1. [1]
    Boyd & Vandenberghe §4.1.1 ↗Boyd & Vandenberghe §4.1.1 · Article

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Objective functions.” Math behind ML. /learn/o-objective