Attention, derived rather than quoted
Most explanations hand you softmax(QKᵀ/√d) and move on. Here is where each piece comes from, why the scaling factor is a square root, and what breaks without it.
Every explanation of attention starts by writing down
Attention(Q, K, V) = softmax(QKᵀ / √d_k) · Vand then explains what the letters mean. That is backwards. The formula is the answer to a question, so let us start with the question.
The question
You have a sequence of token representations. For a given position, you want a representation that mixes in information from other positions — but not uniformly, and not by fixed position. The mixing weights should depend on content.
So you need three things: a way for a position to describe what it is looking for, a way for every position to advertise what it offers, and the thing that actually gets mixed. Those are Q, K and V. They are not mystical; they are three learned projections of the same input because those three roles are different.
Why a dot product
You need a scalar compatibility score between a query and a key. The dot product is the cheapest bilinear form available, it is differentiable everywhere, and one matrix multiply computes all n² of them at once. That last property is the whole reason transformers train faster than RNNs — every score is independent, so the GPU does them in parallel.
Where the √d_k comes from
This is the step usually asserted. Suppose the components of q and k are independent with mean 0 and variance 1. Then
q · k = Σ qᵢkᵢ for i = 1..d_kEach term has variance 1, and variance adds over independent terms, so
Var(q · k) = d_k SD(q · k) = √d_kWith d_k = 64, scores land in a range of roughly ±8 rather than ±1. Feed those to softmax and you get a near one-hot distribution — the maximum swamps everything. And a saturated softmax has a vanishing gradient, so the layer stops learning.
Dividing by √d_k restores unit variance, which keeps softmax in its responsive range. That is the entire justification: it is variance normalisation, not a tuned constant.
What multi-head actually buys
One attention distribution per position forces one notion of relevance. Multi-head splits d_model into h subspaces and runs an independent head in each, so one head can track syntactic agreement while another tracks coreference. Concatenating and projecting recombines them.
Note the parameter count barely changes: h heads of dimension d_model/h cost what one head of dimension d_model costs. You buy specialisation, not capacity.
Where the intuition breaks
Attention weights are not explanations. A head with high weight on a token has not necessarily used that token — the value vector it pulls may be near zero. Reading attention maps as reasoning traces is a mistake, and one that a lot of published figures make.
— Ishaan Sandhwar