Subgradients

01Understand the idea#

Ordinary derivatives require one local slope, but penalties such as absolute value have a corner at zero. Convex subgradients extend slope information to that case. A valid subgradient gives a line touching the function at the current point and staying below it everywhere.

A convex subgradient gives a supporting line below the function. At x, slope g must satisfy this for every z:

At a smooth point it is the derivative. At a kink there can be an interval of valid slopes.

For |x| at zero, slopes from −1 through 1 all work. Slope 0.25 is valid; slope 3 fails at z=1 because 1<3.

Link to this figure ↗Download SVGDownload PNGCopy linkCopy embed

At zero, $f(x)=|x|$ has right slope 1 and left slope $-1$. A line $gz$ stays below $|z|$ for every $z$ exactly when $-1\le g\le1$. If $g=3$, try $z=1$: the line gives 3 above the function’s 1, so that proposed slope fails.

02For |x|, the slope is −1 when x<0 and +1 when x>0#

For |x|, the slope is −1 when x<0 and +1 when x>0. At zero the full set is [−1,1].

For a|x−c| with a>0, the kink moves to c and its slope interval becomes [−a,a].

03Checking z=0 alone cannot reject any proposed slope at the origin: both sides equal zero#

Checking z=0 alone cannot reject any proposed slope at the origin: both sides equal zero. Use a point that can violate the global supporting-line condition.

A slope inside [−1,1] needs no repair for |x| at zero. Nondifferentiability does not mean there is no useful first-order information.

04This supporting-line view explains how optimization can work with a nonsmooth L1 penalty#

This supporting-line view explains how optimization can work with a nonsmooth L1 penalty. A single point on the line does not certify it; the condition must hold everywhere. At a smooth point of a convex function, the ordinary derivative supplies its subgradient.