A retrieval-augmented generation system almost never fails loudly. It does not throw, it does not time out, and it does not return an empty response. It returns a fluent, well-structured, entirely wrong answer with a citation attached — which is considerably worse than an error, because nobody files a bug report against a paragraph that reads well.
Across the RAG systems we have taken to production, the failure is almost never the language model. It is the retrieval step handing over the wrong context, and the model doing exactly what it was asked to do with it.
Start by separating retrieval from generation
The first thing we do on any struggling pipeline is stop evaluating the final answer. It conflates two independent systems. Instead, evaluate them separately, because the fixes are completely different.
- Retrieval quality: for a set of known questions, is the correct source chunk in the top-k results at all? If it is not, no amount of prompt engineering will save you.
- Generation faithfulness: given the correct chunk, does the model actually use it, or does it fall back on its own pretrained assumptions?
In our experience roughly four out of five reported RAG failures are retrieval failures. Teams spend weeks tuning the prompt for a problem that lives two steps upstream.
If the right chunk never reaches the context window, you are not debugging an AI problem. You are debugging a search problem that happens to have a language model attached to it.
The chunking strategy is usually the culprit
Most pipelines start with fixed-size chunking — split every document at 512 tokens, add a little overlap, ship it. It is a reasonable default and a terrible endpoint. Fixed-size splitting cuts tables in half, separates a heading from the paragraph it introduces, and strands a definition away from the term being defined.
What has consistently worked better for us is chunking along the document's own structure, then enriching each chunk with the context it lost:
- Split on semantic boundaries — headings, sections, list groups — rather than a token count.
- Prepend the document title and the heading path to every chunk before embedding it, so an isolated paragraph still carries its topic.
- Keep tables and code blocks intact even when they exceed your target size. A truncated table is worse than a large chunk.
- Store a wider parent window alongside each chunk, retrieve on the small one, and pass the larger one to the model.
That last pattern — small-to-big retrieval — tends to give the biggest single improvement for the least work. You match precisely and answer with full context.
Pure vector search is not enough on its own
Embeddings are excellent at conceptual similarity and surprisingly poor at exact identifiers. Ask about error code E-4021 or a specific SKU and a dense vector search will cheerfully return the twenty chunks that talk about error codes in general, none of which mention that one.
Hybrid retrieval fixes this. Run a keyword search such as BM25 alongside the vector search, then merge the two ranked lists — reciprocal rank fusion is a reasonable default and needs no tuning. Keyword search catches the exact strings; vector search catches the paraphrases. Neither is sufficient alone.
Above that, a cross-encoder reranker over the merged top 50 typically buys another solid accuracy jump. It is slower per document than the initial retrieval, which is exactly why it runs on 50 candidates rather than 50,000.
Build the evaluation set before you tune anything
This is the step teams skip, and skipping it means every subsequent change is a guess. You need somewhere between 50 and 200 real questions with known correct source documents. Pull them from support tickets, internal search logs, or a two-hour session with the people who will actually use the system.
With that set in place you can measure recall at k on every retrieval change and know within minutes whether a new chunking strategy helped or hurt. Without it you are shipping vibes.
Make the model allowed to say no
Even with retrieval fixed, some questions have no answer in the corpus. A model instructed to be helpful will invent one. The fix is a prompt that makes abstention an explicitly acceptable outcome, plus a relevance threshold below which you do not call the model at all and return a clean 'not found' instead.
Users forgive a system that admits its limits. They do not forgive one that confidently misquotes a policy document.
The short version
- Measure retrieval and generation separately — most failures are retrieval.
- Chunk on structure, enrich with heading context, retrieve small and answer big.
- Run hybrid keyword plus vector search, then rerank the merged candidates.
- Build a 50–200 question evaluation set before tuning a single parameter.
- Let the system return nothing when it knows nothing.
None of this is exotic. It is careful information retrieval with a language model on the end, and treating it that way is most of the fix.
