Ridge solutions

01Ridge regression trades some data fit for smaller coefficient magnitudes#

Ridge regression trades some data fit for smaller coefficient magnitudes. This can stabilize a fit when feature columns are nearly redundant. The penalty changes the equations used to solve for coefficients, and its exact effect depends on which coefficients are penalized.

For a half-squared residual sum plus λ||β||²/2, λ≥0, penalizing every coefficient gives:

Use a stated diagonal mask instead of I when some coefficients are unpenalized. We write G=XᵀX and r=Xᵀy.

If the intercept is first and unpenalized, add no penalty to its diagonal. With G=diag(4,2), r=(8,6) and λ=2, solve diag(4,4)β=r to get (2,1.5).

With $G=X^TX=\operatorname{diag}(4,2)$, right side $(8,6)$, and $\lambda=2$, an unpenalized first intercept uses mask $\operatorname{diag}(0,1)$. Add the penalty to get diagonal $(4,4)$. Solve $4\beta_0=8$ and $4\beta_1=6$, giving $(2,1.5)$. The slope shrinks from 3; the intercept equation stays the same.

02An intercept often remains unpenalized, but the objective decides#

An intercept often remains unpenalized, but the objective decides. A mask diag(0,1) leaves the first coefficient alone while penalizing the second.

Apply the stated parameter order. An unpenalized intercept does not mean all coefficients escape the penalty.

03The identity penalty changes diagonal entries only#

The identity penalty changes diagonal entries only. Existing off-diagonal terms still couple coefficient equations.

Solve that coupled system as a system; dividing each right-hand side entry by its diagonal would ignore the coupling.

04An identity penalty adds to diagonal entries, not every matrix entry#

An identity penalty adds to diagonal entries, not every matrix entry. When off-diagonal terms are present, keep them and solve the coupled system. The penalty can make coefficients less sensitive, but its strength should be chosen using a valid training-and-validation procedure.