Bias, variance, and what L1 and L2 actually do to the weights

“Underfitting is bias, overfitting is variance” is where most explanations stop. The decomposition is short enough to derive, and deriving it makes the regularisation choices obvious rather than memorised.

The decomposition

Assume the data is generated with irreducible noise:

y = f(x) + ε        E[ε] = 0,  Var(ε) = σ²

f̂ is the model fitted on a random training set, so it is itself random. Expected squared error at a fixed x, over both the noise and the draw of the training set:

E[(y - f̂)²] = E[(f + ε - f̂)²]
            = E[(f - f̂)²] + E[ε²] + 2E[ε(f - f̂)]
            = E[(f - f̂)²] + σ² + 0          ε independent of the fit

Now split the first term by adding and subtracting E[f̂], the average prediction over training sets:

E[(f - f̂)²] = E[(f - E[f̂] + E[f̂] - f̂)²]
            = (f - E[f̂])²  +  E[(f̂ - E[f̂])²]
              ─────────────     ────────────────
                 Bias²             Variance

The cross term vanishes because E[f̂ - E[f̂]] = 0. So:

Expected error = Bias² + Variance + σ²

Three parts, and only two of them are yours. Bias is being wrong on average — the model cannot represent f. Variance is being unstable — refit on a different sample and the prediction moves. σ² is noise; no model beats it, which is why 100% accuracy on a noisy label set means a leak, not a triumph.

Reading it off a training curve

Symptom	Diagnosis	What helps
Train error high, val error ≈ train	bias	bigger model, better features, train longer
Train error low, val error much higher	variance	more data, regularisation, smaller model
Both low, test error high	leakage or shift	fix the split before touching the model

More data reduces variance and does nothing for bias. That single fact decides whether collecting another 10,000 rows is worth the week it costs.

L2 (ridge): shrink everything

Add the squared norm of the weights to the loss:

J(w) = ‖Xw - y‖² + λ‖w‖²

∂J/∂w = 2Xᵀ(Xw - y) + 2λw = 0

w = (XᵀX + λI)⁻¹Xᵀy

Two consequences fall straight out of that expression. (XᵀX + λI) is invertible for any λ > 0 even when XᵀX is singular — ridge fixes collinearity as a side effect. And the gradient step becomes

w ← w(1 - 2ηλ) - η·(data gradient)

which is literally weight decay: every weight is multiplied by a factor slightly below 1 each step. Shrinkage toward zero, never exactly zero.

L1 (lasso): set things to exactly zero

J(w) = ‖Xw - y‖² + λ‖w‖₁          ‖w‖₁ = Σ|wᵢ|

|w| is not differentiable at 0, and that kink is the entire point. Away from zero its derivative is sign(w), so the penalty pushes by a constant λ regardless of how small the weight is:

L2 penalty gradient:  2λw     → vanishes as w → 0, so w only approaches 0
L1 penalty gradient:  λ·sign(w) → constant push, so w reaches 0 and stops

Once a coefficient hits zero, any move away costs more penalty than it saves in fit — the subgradient condition — so it stays there. That is why lasso performs feature selection and ridge does not.

Geometrically: the L1 constraint region is a diamond with corners on the axes, and a corner is where the loss contour is most likely to first touch. The L2 region is a circle, which has no corners to touch.

Choosing between them

• Ridge when features are correlated and you want them all kept and stabilised. It splits weight between correlated features.
• Lasso when you suspect most features are useless and want a short list. It picks one from a correlated group, arbitrarily.
• Elastic net (λ₁‖w‖₁ + λ₂‖w‖²) when both apply, which in practice is often.

λ is chosen by cross-validation, never by looking at test error — the moment you tune on test, test stops being an estimate of anything.

from sklearn.linear_model import RidgeCV, LassoCV
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler

# Scaling is not optional here: both penalties are scale-dependent, so a
# feature measured in millimetres would otherwise be penalised differently
# from the same feature measured in metres.
model = Pipeline([
    ("scale", StandardScaler()),
    ("ridge", RidgeCV(alphas=[0.01, 0.1, 1.0, 10.0], cv=5)),
])

What to take forward

• Error = Bias² + Variance + noise. Only the first two are under your control.
• More data attacks variance only.
• L2 shrinks smoothly and fixes collinearity; L1 zeroes coefficients outright.
• The difference is the penalty’s gradient near zero: vanishing versus constant.
• Always scale features before penalising them.

---

— Ishaan SandhwarRegularisationModel selectionStatistics