In a RAG system, retrieval is the model
Three weeks of fine-tuning bought six points. Two days of fixing chunking bought nineteen. Here is why that keeps happening.
The mental model most people bring to retrieval-augmented generation is that the LLM is the system and retrieval feeds it. For anything with a real corpus, that is backwards.
What actually happened
On the domain assistant I built, fine-tuning the generator with QLoRA on 4k curated pairs bought about 6 points of answer accuracy over three weeks of work. Replacing fixed-size chunking with semantic-boundary chunking bought about 19 points in two days.
Why the asymmetry
A generator can only be as right as its context. If the retrieved span does not contain the answer, no amount of fine-tuning invents it — you have trained the model to be more fluent about being wrong.
Retrieval failure is unrecoverable downstream. Generation failure is not.
The three chunking mistakes
Fixed token windows. Splitting at 512 tokens cuts tables in half and separates a claim from the qualifier that limits it. The retrieved chunk looks relevant and is actively misleading.
No overlap. An answer straddling a boundary is retrievable from neither side.
Discarding structure. Headings, list context and the source span are the cheapest relevance signal available. Keeping the heading path inside the chunk text improved retrieval measurably and cost nothing.
Evaluate the retriever on its own
This is the habit that changed my results most. Build a small set of questions with known answer spans and measure recall@k of the retriever alone, with no generator in the loop. If recall@5 is 0.6, your ceiling is 0.6, and every hour spent on the generator is spent under that ceiling.
Groundedness beats accuracy as a metric
Accuracy on a held-out set hid the failure I actually cared about: confident answers with no supporting span. A check that rejects claims not covered by the retrieved context took unsupported answers from 14% to 3% — and it is a retrieval-side fix, not a model-side one.
— Ishaan Sandhwar