L2 penalties
A λ||w||²/2 penalty contributes λw; a λ||w||² penalty contributes 2λw.
On this page 7 sections
01An L2 penalty with λ≥0 adds squared coefficient size#
An L2 penalty with λ≥0 adds squared coefficient size. Its gradient depends on the written coefficient:
For λ=3 and w=(2,−1), the penalty gradient is (6,−3). The half cancels the derivative’s factor of two.
P(w)=6w²/2. At w=-2, find P′(w).
Show answer and explanation
Gradient factor 6; 6×(-2)=-12.
02Differentiate each squared coordinate separately#
Differentiate each squared coordinate separately. A negative weight contributes a negative gradient component; zero contributes zero.
Without the half, P(w)=λ||w||² has gradient 2λw. Read the written coefficient before computing each component.
P(w)=2||w||₂², w=. Find the full penalty gradient.
Show answer and explanation
The gradient is 4w=.
03The penalty gradient adds to the data-loss gradient#
The penalty gradient adds to the data-loss gradient. It is a vector contribution, not the scalar penalty value.
Read the actual objective before borrowing a memorized regularization factor.
P(w)=7w², w=3. Draft gradient 21 uses the other half-factor convention. Correct it.
Show answer and explanation
Gradient factor 14; 14×3=42.
Check the half-factor and apply the resulting coefficient to every weight coordinate.
- Compute the gradient contribution of a stated quadratic penalty.
Sources & further reading
- [1]D2L §3.7.1 equations 3.7.2–3.7.3 ↗D2L §3.7.1 equations 3.7.2–3.7.3 · Article