Skip to content
Glacius

Builds on gradients

Optimization

Follow the steps that improve a model, from gradient descent to regularization.

Start with objective functions 27 lessons · 5 units
xyθ
Explore gradient descent steps

Inside this subject

Read at your own pace
01Optimization problems5 lessons
  1. 01
    Objective functions

    Express a scalar objective for specified decision variables.

  2. 02
    Feasible sets

    Identify decisions satisfying all stated constraints.

  3. 03
    Minima and minimizers

    Distinguish the best objective value from the decision attaining it.

  4. 04
    Local and global optima

    Distinguish neighborhood optimality from optimality over the feasible set.

  5. 05
    Empirical objectives

    Form an average per-example loss over a fixed dataset.

02Convexity and optimality3 lessons
  1. 01
    Convex sets

    Check whether a line segment between feasible points remains feasible.

  2. 02
    Convex functions

    Check the convexity inequality for a supplied scalar example.

  3. 03
    First-order optimality

    Apply the zero-gradient necessary condition to a differentiable interior local optimum.

03Gradient methods5 lessons
  1. 01
    Descent directions

    Check whether a direction has a negative first-order objective change.

  2. 02
    Gradient descent steps

    Compute one gradient descent update.

  3. 03
    Learning rates

    Diagnose overshooting from a step size on a one-dimensional quadratic.

  4. 04
    Stopping criteria

    Assess a proposed stopping decision using gradient size and supplied tolerances.

  5. 05
    Momentum

    Compute a momentum update using a stated velocity convention.

04Stochastic optimization8 lessons
  1. 01
    Sample gradients

    Compute a single-example gradient for an empirical objective.

  2. 02
    Minibatch gradients

    Average a minibatch's per-example gradients.

  3. 03
    Gradient noise

    Interpret batch-size effects under a stated independent-gradient variance model.

  4. 04
    Stochastic gradient descent

    Compute a parameter update using a supplied sampled gradient.

  5. 05
    Adaptive scaling

    Compute an update with supplied per-coordinate second-moment scaling.

  6. 06
    Adam updates

    Compute an Adam parameter update from supplied bias-corrected moment estimates.

  7. 07
    Moment bias correction

    Correct a zero-initialized exponential moving average using a supplied formula.

  8. 08
    Gradient clipping

    Rescale a gradient to satisfy a supplied norm threshold.

05Regularization and curvature6 lessons
  1. 01
    L2 penalties

    Compute the gradient contribution of a stated quadratic penalty.

  2. 02
    Ridge solutions

    Solve a small penalized least-squares system with a stated treatment of the intercept.

  3. 03
    L1 penalties

    Compute an L1-penalized objective for supplied coefficients.

  4. 04
    Subgradients

    Identify valid subgradients of absolute value, including at zero.

  5. 05
    Soft thresholding

    Apply a supplied soft-thresholding operator to one scalar coordinate.

  6. 06
    Proximal gradient steps

    Compute a supplied proximal-gradient step for an L1-regularized objective.