Fine-tuning a 7B model on a 12GB card
The memory arithmetic that decides whether a fine-tune fits, and the four settings that actually move the number. No A100 required.
The question is never “can I fine-tune this model”, it is “does it fit”. That is arithmetic, so let us do the arithmetic.
Where the memory goes
Full fine-tuning of a 7B model in fp16 needs, per parameter:
| Item | Bytes/param | 7B total |
|---|---|---|
| Weights (fp16) | 2 | 14 GB |
| Gradients (fp16) | 2 | 14 GB |
| Adam state (m, v, fp32) | 8 | 56 GB |
| Subtotal | 12 | 84 GB |
Plus activations. On a 12 GB card this is not a tuning problem, it is a factor of seven.
What QLoRA changes
Three things, in order of impact.
Quantise the frozen base to 4-bit. 14 GB drops to 3.5 GB. The base never receives gradients, so quantisation error only affects the forward pass.
Train adapters, not weights. LoRA replaces the update ΔW with a low-rank product BA, where B is d×r and A is r×k. At r = 16 on a 7B model that is roughly 20M trainable parameters instead of 7B — a 350× cut — and gradients plus optimiser state shrink with it, to well under 1 GB.
Page the optimiser. Paged AdamW spills optimiser state to CPU memory on the spikes that would otherwise OOM at the worst possible moment.
Running total: about 4.5 GB before activations, which leaves real headroom on a 12 GB card.
The four settings that matter
- Rank. Start at 16. Below 8, quality drops. Above 32 you are mostly fitting noise unless the dataset is large.
- Target modules. Adapting the attention projections alone is the common default. Including the MLP projections costs more memory and tends to help on tasks needing new knowledge rather than a new output format.
- Alpha. The effective scaling is α/r. Keeping α = 2r is a reasonable default — and note that changing r without changing α silently changes your effective learning rate.
- Sequence length. Activation memory grows with it. Halving max length is often the cheapest escape from an OOM.
The flag people skip
Gradient checkpointing trades roughly 30% more compute for a large activation saving. On a card this size that trade is almost always worth taking, and it is one argument.
Result
7B parameters, a single 12 GB consumer card, about 10 GB peak, and evaluation within roughly two points of the same recipe at full precision. The bottleneck moves back where it belongs: the quality of your data.
— Ishaan Sandhwar