Skip to content
Glacius
OptimizationConcept reference

L1 penalties

An L1-penalized objective adds λ times the absolute-value sum of its penalized coefficients to data loss.

1 min read1 figureSources & further reading
On this page 7 sections
  1. Overview
  2. An L1 penalty adds the absolute sizes of the penalized coefficients
  3. A candidate with lower data loss can still have a higher penalized objective
  4. Only explicitly penalized coefficients belong in the sum
  5. Key takeaway
  6. Sources & further reading
  7. Concept connections

01An L1 penalty adds the absolute sizes of the penalized coefficients#

An L1 penalty adds the absolute sizes of the penalized coefficients. Positive and negative coefficients both add cost:

J=L+λ∑j∣wj∣J=L+\lambda\sum_j|w_j|

L is the data loss and λ is a nonnegative penalty weight.

For loss 3, w=(−2,1) and λ=2, the absolute values sum to 3. Penalty 6 gives a total objective of 9.

For data loss 3, weights (−2,1) and lambda 2, L1 norm is 3 and the penalty is 6. Proportional segments total objective 9. The regularizer uses magnitudes, so the negative weight contributes positively.For data loss 3, weights (−2,1) and lambda 2, L1 norm is 3 and the penalty is 6. Proportional segments total objective 9. The regularizer uses magnitudes, so the negative weight contributes positively.
Figure 1For data loss 3, weights (−2,1) and lambda 2, L1 norm is 3 and the penalty is 6. Proportional segments total objective 9. The regularizer uses magnitudes, so the negative weight contributes positively.
Link to this figure ↗Download SVGDownload PNG
Check your reasoning

J=L+λ||w||₁. Given L=2, λ=2, w=(−2,3)(-2, 3). Find J.

Show answer and explanation
12

Absolute values sum to 5; total 12.

02A candidate with lower data loss can still have a higher penalized objective#

A candidate with lower data loss can still have a higher penalized objective. Compute loss plus penalty for each candidate using the same λ.

The result is one scalar score. Coefficient sign cancellation would incorrectly erase part of the penalty.

Check your reasoning

Compare J=L+λ||w||₁, λ=1. A: L=2, w=(3,0)(3, 0). B: L=3, w=(1,0)(1, 0). Lower J?

  1. AA
  2. BB
  3. CTie
Show answer and explanation
B

J_A=5, J_B=4.

03Only explicitly penalized coefficients belong in the sum#

Only explicitly penalized coefficients belong in the sum. If the intercept is excluded, its size must not affect this penalty.

Absolute value also has a kink at zero. L1 penalties are not differentiable everywhere; a separate treatment handles that point.

Check your reasoning

J=L+λ||w||₁; intercept b is excluded. L=4, λ=2, b=-3, slopes w=(2,−1)(2, -1). Draft J=16 includes b. Correct J.

Show answer and explanation
10

Only slopes are penalized; J=10.

Key takeaway

Take absolute values, respect excluded coefficients, multiply by λ and add the data loss.

  • Compute an L1-penalized objective for supplied coefficients.

Sources & further reading

  1. [1]
    Parikh & Boyd §7.1 Lasso and §6.5.2 ↗Parikh & Boyd §7.1 Lasso and §6.5.2 · Article

Reference this concept

Link to this page, a section, or an individual figure.

Glacius. “L1 penalties.” Math behind ML. /learn/o-lone-penalty