fig. 03notes
Working it out in public
A path, not an archive: it starts at the maths you need before any of it makes sense and ends at the papers. If I cannot derive something, I have not understood it — so these are the derivations, written at the point I finally got them.
topic
- attention weightsShapes first — vectors, matrices, and why matmul is the whole gameThe linear algebra you need before any of the rest makes sense, written as the shape rules I check before running anything.
- training lossFrom one derivative to gradient descent, with numbersWhat a derivative actually tells you, why the update rule subtracts it, and the exact learning rate at which the whole thing blows up.
- embedding spaceLinear regression two ways — closed form and gradient descentDeriving the normal equation from the gradient, solving a three-point dataset by hand, and the cost analysis that says when to give up on the exact answer.
- episode returnLogistic regression, and why the gradient is just (p − y)Deriving the sigmoid's derivative, the cross-entropy loss, and the one-line gradient that falls out — plus why squared error is the wrong loss for classification.
- per-class APSoftmax without overflow — the max-subtraction trick, provedWhy exp() overflows at around 709, why subtracting the max changes nothing mathematically, and how log-sum-exp keeps the whole thing finite.
- service graphBackpropagation by hand — one forward pass, one backward pass, real numbersA two-layer network worked end to end with arithmetic you can check, including the update and the loss afterwards to prove it went down.
- training lossBias, variance, and what L1 and L2 actually do to the weightsThe decomposition derived in four lines, then why ridge shrinks everything and lasso sets coefficients to exactly zero.
- embedding spaceThe split is the experiment — five ways data leaksWhy fitting a scaler before splitting inflates your score, plus the group, temporal and duplicate leaks that survive a correct train_test_split.
- service graphThe master theorem, and the recurrences it refuses to solveThe three cases with the comparison that decides them, worked on merge sort, binary search and Strassen — plus the GATE favourite that falls in the gap.
- attention weightsNormalisation up to BCNF, and the dependency you have to give upAttribute closure, candidate keys, and 1NF through BCNF on one small relation — including the standard example where BCNF costs you a functional dependency.
- per-class APYour model is not wrong, it is overconfidentAccuracy tells you how often the label is right. Calibration tells you whether the confidence means anything — and for anything with a human downstream, the second matters more.
- per-class APPage replacement, and the GATE trap in Belady's anomalyFIFO, LRU and Optimal worked end to end on one reference string, plus the question pattern that catches people every year.
- attention weightsAttention, derived rather than quotedMost explanations hand you softmax(QKᵀ/√d) and move on. Here is where each piece comes from, why the scaling factor is a square root, and what breaks without it.
- training lossFine-tuning a 7B model on a 12GB cardThe memory arithmetic that decides whether a fine-tune fits, and the four settings that actually move the number. No A100 required.
- embedding spaceIn a RAG system, retrieval is the modelThree weeks of fine-tuning bought six points. Two days of fixing chunking bought nineteen. Here is why that keeps happening.
Nothing filed under that topic yet.