Ridge vs lasso: why one coefficient becomes zero

A fitted linear model has two coefficients: $3$ and $0.8$. Add a ridge penalty and, in the example below, they become $1.5$ and $0.4$. Add a lasso penalty and they become $2$ and $0$. Both fits accept more training error in exchange for smaller coefficients. Only the second has removed a feature from its prediction.

The difference comes from the penalty near zero. Ridge penalizes squared coefficients, so its pull toward zero weakens as a coefficient gets smaller. Lasso penalizes absolute values and can make zero the exact optimum over a whole range of inputs. That is the basis of lasso's feature selection, although a selected feature is not necessarily the uniquely important one.

We can derive those four numbers from a dataset small enough to inspect, then change the dataset to see where the feature-selection interpretation breaks down. You will need linear regression and basic derivatives; the matrix notation only packages the four rows.

Give both methods the same data

Our synthetic dataset has four observations and two features. The prediction is $\hat y=b_1x_1+b_2x_2$.

These values were constructed as $y=3x_1+0.8x_2$. There is no noise, and ordinary least squares recovers $(3,0.8)$ with zero training error. This is a calculation about penalties, not evidence about predictive performance on new data.

Both feature columns have mean zero and mean square one. Their dot product is zero: knowing one column does not help reconstruct the other through a linear combination. If $X$ is the matrix of feature values and $n=4$, those properties give $X^TX/n=I$. The response is also centered, so the fitted intercept is zero; we leave it out of the calculation.

Use the following objective conventions throughout:

$$J_{\text{ridge}}(b) =\frac{1}{2n}\lVert y-Xb\rVert_2^2 +\frac{\lambda}{2}\sum_j b_j^2,$$ $$J_{\text{lasso}}(b) =\frac{1}{2n}\lVert y-Xb\rVert_2^2 +\lambda\sum_j |b_j|.$$

Here $\lambda\geq0$ controls the penalty. The factor of $1/2$ in the ridge penalty simplifies its derivative. These are standard forms, but libraries differ in how they scale the loss and name the penalty parameter. We will translate them into scikit-learn's alpha below.

Because our columns are orthogonal and scaled, the data-loss term simplifies to

$$\frac{1}{2n}\lVert y-Xb\rVert_2^2 =\frac12(b_1-3)^2+\frac12(b_2-0.8)^2.$$

Each coefficient can now be optimized separately. This separation is a convenience of the example; it does not hold for arbitrary correlated features.

Ridge divides; lasso subtracts until it reaches zero

Write $z$ for one unpenalized coefficient, either $3$ or $0.8$. Its ridge objective is

$$\frac12(b-z)^2+\frac{\lambda}{2}b^2.$$

Setting the derivative to zero gives $b-z+\lambda b=0$, so

$$b_{\text{ridge}}=\frac{z}{1+\lambda}.$$

At $\lambda=1$, ridge halves both coefficients: $(1.5,0.4)$. For either nonzero $z$ in this example, any finite penalty leaves a nonzero result. If $z$ were already zero, ridge would return zero too. “Ridge never produces a zero coefficient” is therefore too strong; it has no lasso-like interval that thresholds small nonzero values to zero.

Lasso replaces the squared penalty with $\lambda|b|$. On the positive side, its derivative is $b-z+\lambda$, which suggests $b=z-\lambda$. That solution works only while it stays positive. On the negative side, the derivative is $b-z-\lambda$, giving $b=z+\lambda$ while that value is negative.

At zero, look at the slopes from each side. The slope just to the left is $-z-\lambda$; just to the right it is $-z+\lambda$. Zero minimizes this convex objective when the left slope is nonpositive and the right slope is nonnegative, which happens exactly when $|z|\leq\lambda$. Combining the cases gives soft thresholding:

$$b_{\text{lasso}} =\operatorname{sign}(z)\max(|z|-\lambda,0).$$

At $\lambda=1$, the coefficient $3$ becomes $2$, while $0.8$ becomes zero. Lasso is solving a different optimization problem, rather than rounding a small ridge coefficient after fitting.

Figure link ↗SVG ↓PNG ↓Data ↓

The regularization paths in the opening figure follow these formulas as $\lambda$ increases. Ridge's coefficients approach zero smoothly. Lasso's smaller coefficient reaches zero at $\lambda=0.8$, and its larger coefficient reaches zero at $\lambda=3$. These simple monotone paths depend on our orthogonal design; individual paths with correlated predictors can behave differently.

Reproduce the coefficients in Python

The scikit-learn Ridge objective is the sum of squared residuals plus alpha times the squared coefficient norm. Multiply our ridge objective by $2n$ and you get that form with alpha = n * lambda.

The scikit-learn Lasso objective already divides the squared residuals by $2n$, so its alpha equals our $\lambda$. Using the same numerical alpha in both estimators would not reproduce the conventions above. Even after this conversion, a shared $\lambda$ is an illustrative setting, not a claim that both models have equally strong regularization or equally good validation performance.

import numpy as np
from sklearn.linear_model import Ridge, Lasso

X = np.array([[1, 1], [1, -1], [-1, 1], [-1, -1]], dtype=float)
y = X @ np.array([3.0, 0.8])
n = len(y)
lam = 1.0

# X and y are already centered; feature mean squares are one.
ridge = Ridge(alpha=n * lam, fit_intercept=False, solver="svd")
lasso = Lasso(alpha=lam, fit_intercept=False, tol=1e-12,
              max_iter=10000)

for name, model in [("ridge", ridge), ("lasso", lasso)]:
    model.fit(X, y)
    mse = np.mean((y - model.predict(X)) ** 2)
    print(f"{name}: coefficients={model.coef_.round(4)}, mse={mse:.4f}")

The output is:

ridge: coefficients=[1.5 0.4], mse=2.4100
lasso: coefficients=[2. 0.], mse=1.6400

Neither training error beats the unpenalized fit, which is exact. Lasso's lower training error at this particular setting does not establish that lasso is the better predictor. Regularization deliberately changes the training objective; its value on unseen data must be checked separately.

A zero coefficient does not settle which feature matters

Now replace the two different columns with two copies of the same centered, unit-variance feature $x$, and let $y=3x$. A prediction uses only the sum of the coefficients:

$$\hat y=b_1x+b_2x=(b_1+b_2)x.$$

At $\lambda=1$, the lasso objective becomes

$$\frac12(3-b_1-b_2)^2+|b_1|+|b_2|.$$

Fix a nonnegative total $s=b_1+b_2$. Every nonnegative split of that total has the same penalty, $|b_1|+|b_2|=s$. Minimizing $\frac12(3-s)^2+s$ gives $s=2$. Thus $(2,0)$, $(1,1)$, and $(0,2)$ all minimize the same lasso objective and produce identical predictions. A solver can return a sparse answer, but the data have not established that one of the duplicate features deserves the credit.

For ridge, a fixed total $s$ has the smallest squared penalty when the coefficients share it equally: $b_1=b_2=s/2$. Substitution gives $\frac12(3-s)^2+s^2/4$, whose minimum is also $s=2$ at this setting. Ridge has the unique solution $(1,1)$.

Figure link ↗SVG ↓PNG ↓Data ↓

Exact duplication makes the ambiguity visible without numerical noise. With strongly correlated but distinct features, coefficients and the selected subset can be sensitive to changes in the data; perfect duplication is not required for selection to deserve scrutiny. The scikit-learn linear-model guide describes elastic net as a combination of L1 and L2 penalties that is useful with correlated features. It offers another candidate to validate, rather than a guarantee of correct feature discovery.

Choose using validation, scaling, and the model you need

Ridge is a useful baseline when you want shrinkage while keeping contributions from many features. Lasso is worth testing when a sparse model has practical value, such as reducing the number of inputs you must collect. Compare them on validation data appropriate to the task, with a separately chosen penalty for each. A coefficient becoming zero is an optimization result; it is not a statistical significance test or proof of a causal relationship.

Scaling belongs inside that comparison. If a feature is multiplied by $100$, its coefficient can be divided by $100$ without changing the prediction. Its absolute-value penalty then becomes $100$ times smaller, and its squared penalty becomes $10{,}000$ times smaller. Leaving such unit differences in place changes which coefficients the penalty makes expensive. StandardScaler is a common way to put varying features on comparable scales, although the right treatment depends on what the inputs mean.

Fit that scaling on training data only. During cross-validation, put preprocessing and the estimator in a pipeline so each fold learns its own scaling; otherwise validation information can leak into training. Keep the test set for the final assessment. The scikit-learn guide to data leakage explains this workflow.

For a numerical look at preprocessing, our normalization and standardization comparison follows one column through both transformations and then transforms an unseen value using the fitted training statistics.

The useful question is whether the resulting predictions, retained inputs, and stability meet your needs. In our four-row example, every zero had an exact mathematical reason. With real data, the same penalty rules apply, but the evidence for choosing a model comes from how it performs beyond the rows that fitted it.