Logistic regression, and why the gradient is just (p − y)

The first time I saw grad = p - y in a classification loop I assumed it was a simplification someone had hard-coded. It is not. It is what the chain rule gives you, and the cancellation is worth doing once by hand.

From score to probability

A linear model produces an unbounded score, the logit:

z = wᵀx + b        z ∈ (-∞, ∞)

A probability has to live in (0, 1), so squash it with the sigmoid:

σ(z) = 1 / (1 + e⁻ᶻ)

σ(-∞) → 0      σ(0) = 0.5      σ(∞) → 1

Its derivative is unusually clean, and the cleanliness is the reason everything else collapses later:

σ'(z) = d/dz (1 + e⁻ᶻ)⁻¹
      = -(1 + e⁻ᶻ)⁻² · (-e⁻ᶻ)
      = e⁻ᶻ / (1 + e⁻ᶻ)²
      = [1/(1 + e⁻ᶻ)] · [e⁻ᶻ/(1 + e⁻ᶻ)]
      = σ(z)(1 - σ(z))

So σ' = σ(1 - σ), maximal at z = 0 (value 0.25) and vanishing at both ends. That vanishing is the saturation problem: a confidently wrong unit learns slowly, because its own derivative gates the update.

The loss

For a binary label y ∈ {0, 1} and prediction p = σ(z), cross-entropy — also called log loss — is

L = -[ y·log p + (1 - y)·log(1 - p) ]

Read it one case at a time. If y = 1, only the first term survives: L = -log p, which is 0 when p = 1 and → ∞ as p → 0. If y = 0, only the second: L = -log(1 - p). Either way the penalty is unbounded for confident mistakes, which is exactly the behaviour you want from a probability.

This is not an arbitrary choice. It is the negative log-likelihood of a Bernoulli model, so minimising it is maximum likelihood estimation.

The cancellation

Differentiate L with respect to the logit z, in two steps:

∂L/∂p = -y/p + (1 - y)/(1 - p)

∂p/∂z = p(1 - p)                    (the sigmoid derivative)

∂L/∂z = [ -y/p + (1-y)/(1-p) ] · p(1 - p)
      = -y(1 - p) + (1 - y)p
      = -y + yp + p - yp
      = p - y

Everything cancels. The gradient of the loss with respect to the score is prediction minus target — the signed error — and by the chain rule the gradient with respect to the weights is

∂L/∂w = (p - y)·x

Predicted 0.9 when the answer was 1? Push by 0.1 in the direction of x. That is the entire update, and it is why a logistic layer is so cheap to train.

Why not squared error

(p - y)² with a sigmoid gives

∂L/∂z = 2(p - y)·p(1 - p)

That extra p(1 - p) is the problem. Take a sample with y = 1 where the model says p = 0.001 — as wrong as it gets:

cross-entropy:  |∂L/∂z| = 0.999          strong signal
squared error:  |∂L/∂z| = 2(0.999)(0.001)(0.999) ≈ 0.002

Squared error nearly zeroes the gradient precisely where the model is most wrong. It is also non-convex in w once composed with the sigmoid, while cross-entropy stays convex. Two independent reasons; either alone would settle it.

Numerical care

Never compute log(σ(z)) as written. For z = -800, σ(z) underflows to exactly 0.0 in float64 and the log is -inf, which poisons every subsequent gradient. Work in logit space instead:

# Wrong: two chances to lose the number
loss = -(y * np.log(sigmoid(z)) + (1 - y) * np.log(1 - sigmoid(z)))

# Right: the stable identity, log(1 + eˣ) computed without overflow
loss = np.logaddexp(0, z) - y * z

In PyTorch the same rule reads: pass logits to BCEWithLogitsLoss, never a sigmoid output to BCELoss. The fused version applies the log-sum-exp trick internally.

Multi-class

Replace the sigmoid with softmax over K logits and the same derivation produces the same result, one-hot vector included:

∂L/∂z = p - y        p, y ∈ ℝᴷ

Which is why CrossEntropyLoss in PyTorch takes raw logits and a class index, and why the backward pass of a classifier is one subtraction.

What to take forward

σ' = σ(1 - σ), which saturates at both ends.
Cross-entropy is the Bernoulli negative log-likelihood, not a heuristic.
∂L/∂z = p - y. Learn the derivation once; the result is used everywhere.
Squared error kills the gradient exactly where the model is most wrong.
Compute losses from logits, never from probabilities.

— Ishaan SandhwarClassificationCross-entropyOptimisation