Soft thresholding

Soft thresholding subtracts τ from magnitude with a zero floor and preserves the sign of surviving values.

2 min read1 figureSources & further reading

On this page 8 sections

Overview
Understand the idea
A closer look
Soft thresholding is not clipping large values to ±τ
The flat zero region is why this operation can create sparse coefficient vectors
Key takeaway
Sources & further reading
Concept connections

01Understand the idea#

Soft thresholding moves a value toward zero by a chosen amount, stopping if it reaches zero. In sparse models, this operation can eliminate small coefficients entirely while reducing larger ones. It treats positive and negative values symmetrically by working with their magnitudes.

Soft thresholding shrinks a scalar’s magnitude by τ, stopping at zero. Let m be the remaining magnitude:

\begin{gathered}m=\max(|z|-\tau,0)\\S_\tau(z)=\operatorname{sign}(z)m\end{gathered}

Use τ≥0 and sign(0)=0.

Input −5 with threshold 2 has magnitude 5. Shrink that magnitude to 3, then restore the negative sign to obtain −3.

Figure 1Soft threshold with tau=2 is S2(z)=sign(z)max(|z|−2,0). The graph is zero throughout [−2,2], then has slope 1 on both outer pieces. Input −5 returns −3.

For input $-5$ and threshold 2, start with magnitude 5. Subtract 2 to get 3, then restore the negative sign to obtain $-3$ . For input $-1$ with the same threshold, $1-2$ is negative, so clamp the remaining magnitude at zero. Do not let shrinkage cross zero and reverse the sign.

Check your reasoning

Sτ(z)=sign(z)max(|z|−τ,0). z=-7, τ=3. Find Sτ(z).

Show answer and explanation

-4

Shrink magnitude to 4; output -4.

02A closer look#

If the output survives and keeps the input’s sign, the threshold equals the drop in magnitude. A nonzero output makes that threshold recoverable.

A zero output instead tells you only that the threshold was at least the input magnitude.

Check your reasoning

Sτ(z)=sign(z)max(|z|−τ,0). Input -8 gives nonzero output -5. Find τ.

Show answer and explanation

Threshold = |input|−|output| = 3.

03Soft thresholding is not clipping large values to ±τ#

Soft thresholding is not clipping large values to ±τ. It subtracts τ from their magnitudes and zeroes small values.

Never shrink past zero and flip the sign. The max with zero prevents that overshoot.

Check your reasoning

Sτ(z)=sign(z)max(|z|−τ,0). z=-2, τ=5. Draft output -2. Correct it.

Show answer and explanation

Shrink magnitude to 0; output 0.

04The flat zero region is why this operation can create sparse coefficient vectors#

The flat zero region is why this operation can create sparse coefficient vectors. It differs from clipping: clipping caps large magnitudes, whereas soft thresholding removes small ones and subtracts a fixed amount from larger ones. The next lesson uses it after a gradient step.

Key takeaway

Compare magnitude with τ; zero it if small, otherwise shrink the magnitude without flipping sign.

Apply a supplied soft-thresholding operator to one scalar coordinate.

Sources & further reading

[1]
Parikh & Boyd §6.5.2 equation 6.9 ↗Parikh & Boyd §6.5.2 equation 6.9 · Article

01Understand the idea#

02A closer look#

03Soft thresholding is not clipping large values to ±τ#

04The flat zero region is why this operation can create sparse coefficient vectors#

Compare magnitude with τ; zero it if small, otherwise shrink the magnitude without flipping sign.

Sources & further reading

Reference this concept