Skip to content
Glacius
OptimizationConcept reference

Stopping criteria

A stopping decision compares specified gradient and progress metrics with their supplied tolerances.

1 min read1 figureSources & further reading
On this page 7 sections
  1. Overview
  2. A stopping rule makes a numerical decision using stated tolerances
  3. A combined rule may require several conditions at once
  4. A tiny parameter change may come from a tiny learning rate
  5. Key takeaway
  6. Sources & further reading
  7. Concept connections

01A stopping rule makes a numerical decision using stated tolerances#

A stopping rule makes a numerical decision using stated tolerances. A gradient-norm rule checks how close the current gradient is to zero.

Use the specified norm and comparison sign. Equality passes a ≤ rule but fails a < rule.

For gradient (0.03,0.04), the Euclidean norm is 0.05. A ≤0.05 rule stops at this boundary.

Gradient (.03,.04) has norm .05. The circle is the boundary of the closed norm ball ||g||≤.05; the gradient endpoint lies exactly on it. The stated inclusive stopping rule is satisfied, but this is not a global-optimum certificate.Gradient (.03,.04) has norm .05. The circle is the boundary of the closed norm ball ||g||≤.05; the gradient endpoint lies exactly on it. The stated inclusive stopping rule is satisfied, but this is not a global-optimum certificate.
Figure 1Gradient (.03,.04) has norm .05. The circle is the boundary of the closed norm ball ||g||≤.05; the gradient endpoint lies exactly on it. The stated inclusive stopping rule is satisfied, but this is not a global-optimum certificate.
Link to this figure ↗Download SVGDownload PNG
Check your reasoning

Stop exactly when Euclidean ∥g∥≤0.15\|g\|\le0.15. Given g=(0.09,0.12)g=(0.09,0.12): stop?

  1. AContinue.
  2. BStop.
  3. CNeed loss value.
Show answer and explanation
Stop.

Norm equals tolerance.

02A combined rule may require several conditions at once#

A combined rule may require several conditions at once. If it says AND, every comparison must pass.

Check the stated metrics directly. Small loss change and small gradient norm are different facts.

Check your reasoning

Stop exactly when gradient norm ≤0.02\le0.02 AND loss change ≤0.003\le0.003. Given norm 0.020.02, change 0.0040.004: stop?

  1. AContinue.
  2. BStop.
  3. CEither suffices.
Show answer and explanation
Continue.

Change 0.004>0.003.

03A tiny parameter change may come from a tiny learning rate#

A tiny parameter change may come from a tiny learning rate. It does not establish a small gradient or proximity to a global optimum.

Even satisfying a gradient tolerance is a stopping decision. Global-optimality guarantees need additional assumptions.

Check your reasoning

Stop exactly when gradient norm≤0.02; norm is 0.01. Report: rule met, global optimum unproved. Verdict?

  1. ARule not met.
  2. BGlobal proof.
  3. CReport justified.
Show answer and explanation
Report justified.

A rule, not a proof.

Key takeaway

Apply every required comparison exactly, and keep a stopping decision separate from a global-optimum certificate.

  • Assess a proposed stopping decision using gradient size and supplied tolerances.

Sources & further reading

  1. [1]
  2. [2]
    Dive into Deep Learning §12.3.1.2 ↗Dive into Deep Learning · Article

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “Stopping criteria.” Math behind ML. /learn/o-stopping