From one derivative to gradient descent, with numbers
What a derivative actually tells you, why the update rule subtracts it, and the exact learning rate at which the whole thing blows up.
Every training loop in machine learning is one line — subtract a bit of the gradient — repeated. This note builds that line from a single derivative, then runs it by hand until the number moves.
A derivative is a local slope
For a function of one variable, the derivative at a point is the slope of the tangent there:
f(w) = (w - 3)²
f'(w) = 2(w - 3)At w = 0, f'(0) = -6. A negative slope means the function falls as w
grows, so to make the loss smaller you should move w up. That sign flip is
the whole idea, and it is why the update subtracts:
w ← w - η · f'(w)with η (eta) the learning rate. Slope negative → subtracting it moves w up.
Slope positive → moves it down. Slope zero → nothing happens, which is what a
minimum feels like from the inside.
Run it
Start at w = 0, η = 0.1, minimum at w = 3:
step 0: w = 0.000 f' = -6.000 w ← 0.000 - 0.1(-6.000) = 0.600
step 1: w = 0.600 f' = -4.800 w ← 0.600 + 0.480 = 1.080
step 2: w = 1.080 f' = -3.840 w ← 1.080 + 0.384 = 1.464
step 3: w = 1.464 f' = -3.072 w ← 1.464 + 0.307 = 1.771
...
step 20: w = 2.965 loss = 0.0012Each step closes 20% of the remaining distance. That is not a coincidence —
substitute the error e = w - 3 into the update:
e ← e - η·2e = e(1 - 2η)With η = 0.1 the error is multiplied by 0.8 every step. Geometric decay, and
the entire convergence question is now visible in one factor.
The learning rate has a hard ceiling
Convergence needs |1 - 2η| < 1, i.e. 0 < η < 1. Watch η = 1.1, where the
factor is 1 - 2.2 = -1.2:
w = 0.00 -> 6.60 -> -1.92 -> 10.10 -> -6.42 diverging, sign flippingThe overshoot is larger than the error it was correcting, every time. So when a
loss curve oscillates and grows, that is not bad initialisation and not bad
data — that is η past the ceiling. Divide it by ten before changing anything
else.
For a general quadratic with curvature L (the second derivative), the same
argument gives η < 2/L. Sharper curvature, smaller allowed step — which is why
a learning rate does not transfer between architectures.
More than one parameter: the gradient
With two parameters you take partial derivatives — differentiate with respect to one, hold the other constant:
f(w₁, w₂) = w₁² + 3w₂²
∂f/∂w₁ = 2w₁
∂f/∂w₂ = 6w₂
∇f = [2w₁, 6w₂]The gradient ∇f is those partials stacked into a vector, and it points in the direction of steepest increase. Descent walks the other way:
w ← w - η∇fSame line, now vector-valued. A network with seven million parameters uses this exact update; the only hard part is computing ∇f, which is what backpropagation does.
Note the asymmetry above: w₂ has six times the curvature of w₁, so a step
size that is safe for w₂ crawls for w₁. That mismatch — ill-conditioning —
is the problem momentum and Adam exist to paper over.
Batch, stochastic, mini-batch
The loss is an average over the dataset, so the gradient is an average too:
| Variant | Gradient computed on | Per-step cost | Behaviour |
|---|---|---|---|
| Batch | all n samples |
O(n) | smooth, slow, exact |
| Stochastic | 1 sample | O(1) | noisy, cheap, escapes shallow minima |
| Mini-batch | B samples (32–512) |
O(B) | the practical default |
Mini-batch wins for a reason that is hardware rather than mathematics: a batch of 256 is one matrix multiply on a GPU, and 256 separate samples are 256 of them.
What to take forward
- Derivative = slope; descent subtracts it, which is why the sign works out.
- On a quadratic the error decays by
(1 - 2η)per step — convergence is one factor. - A loss that oscillates and grows means
ηis too large. Nothing else looks like that. - ∇f is partials stacked. It points uphill, so descent negates it.
— Ishaan Sandhwar