Backpropagation by hand — one forward pass, one backward pass, real numbers
A two-layer network worked end to end with arithmetic you can check, including the update and the loss afterwards to prove it went down.
loss.backward() is one line, and for a while I could use it without being able
to reproduce it. This is the note I wrote to fix that: every number below can be
checked on paper.
The network
Two inputs, two hidden units with ReLU, one sigmoid output, binary cross-entropy. Biases start at zero for the hidden layer.
x = [1, 2] target y = 1
W1 = [[ 0.1, -0.2], b1 = [0, 0]
[ 0.3, 0.4]]
W2 = [0.5, -0.6] b2 = 0.1Forward
z1 = W1x = [0.1(1) + (-0.2)(2), 0.3(1) + 0.4(2)]
= [-0.3, 1.1]
a1 = ReLU(z1) = [0, 1.1] first unit is dead for this input
z2 = W2·a1 + b2 = 0.5(0) + (-0.6)(1.1) + 0.1
= -0.56
p = σ(-0.56) = 1 / (1 + e^0.56) = 1 / 2.7506 = 0.3636
L = -log(0.3636) = 1.0117The model says 36% for a class that is actually present. It should push p up.
Backward
Work right to left, carrying ∂L/∂(this node).
Output layer. From the cross-entropy derivation, the gradient with respect to the logit is prediction minus target:
δ2 = ∂L/∂z2 = p - y = 0.3636 - 1 = -0.6364Negative, meaning: increasing z2 decreases the loss. Now distribute it. The
local rule for z2 = W2·a1 + b2 is that each weight’s gradient is δ2 times
the activation it multiplied:
∂L/∂W2 = δ2 · a1 = [-0.6364(0), -0.6364(1.1)] = [0, -0.7000]
∂L/∂b2 = δ2 = -0.6364
∂L/∂a1 = δ2 · W2 = [-0.6364(0.5), -0.6364(-0.6)] = [-0.3182, 0.3818]Note the dead unit: a1₁ = 0, so ∂L/∂W2₁ = 0. A ReLU that is off contributes
no gradient to the weight feeding it — that is the mechanism behind dying ReLUs,
seen from one step away.
Hidden layer. ReLU’s derivative is 1 where z > 0 and 0 where z < 0:
ReLU'(z1) = [0, 1] z1 = [-0.3, 1.1]
δ1 = ∂L/∂a1 ⊙ ReLU'(z1) = [-0.3182(0), 0.3818(1)] = [0, 0.3818]The gradient through the first unit is cut off entirely — this is the “gate”
part of a gated activation. Then the same local rule as before, now with x:
∂L/∂W1 = δ1 ⊗ x = [[0(1), 0(2) ], = [[0, 0 ],
[0.3818(1), 0.3818(2)]] [0.3818, 0.7636]]
∂L/∂b1 = δ1 = [0, 0.3818]Update
Gradient descent with η = 0.1:
W2 ← [0.5, -0.6] - 0.1[0, -0.7000] = [0.5, -0.5300]
b2 ← 0.1 - 0.1(-0.6364) = 0.1636
W1 row 2 ← [0.3, 0.4] - 0.1[0.3818, 0.7636] = [0.2618, 0.3236]
W1 row 1 unchanged — its gradient was zeroDid it work?
Re-run the forward pass with the updated weights:
z1 = [-0.3, 0.2618(1) + 0.3236(2)] = [-0.3, 0.9091]
a1 = [0, 0.9091]
z2 = -0.5300(0.9091) + 0.1636 = -0.3182
p = σ(-0.3182) = 0.4211
L = -log(0.4211) = 0.8648Loss 1.0117 → 0.8648, prediction 0.3636 → 0.4211. One step, in the right
direction, with no framework involved.
The pattern worth memorising
Every layer does the same two things in the backward pass:
- Local gradient — how this layer’s output changes with its input
(
Wfor a linear layer,σ(1-σ)for a sigmoid, a 0/1 mask for ReLU). - Multiply by what came from the right, then pass the result left.
That is the chain rule, arranged so each layer needs to know nothing about its
neighbours beyond one incoming vector. It is also why the cost of the backward
pass is roughly twice the forward pass, and why activations must be kept in
memory until backward runs — ∂L/∂W2 needed a1, which was computed on the way
forward.
Checking your own implementation
Gradient checking, for when the analytic gradient is suspect:
def numeric_grad(f, w, i, eps=1e-5):
"""Central difference for one coordinate of w."""
up, down = w.copy(), w.copy()
up[i] += eps
down[i] -= eps
return (f(up) - f(down)) / (2 * eps)Compare against the analytic value with a relative error test; anything under
about 1e-7 is agreement. Use the central difference, not the forward
difference — its error is O(eps²) rather than O(eps).
What to take forward
δ = p - yat the output, then multiply-and-pass-left all the way down.- A weight’s gradient is the incoming delta times the activation it multiplied.
- ReLU gates the gradient: off means zero, which is how units die.
- Activations are kept because the backward pass needs them.
— Ishaan Sandhwar