RAG: Retrieval-Augmented Generation

The name comes from a 2020 paper by Lewis and colleagues at Facebook AI Research, which paired a neural retriever over Wikipedia with a text generator and showed that the pair answered knowledge-heavy questions better than a much larger model working from memory. Once ChatGPT made LLMs mainstream at the end of 2022, the same idea became the default way to point a model at private or changing data: company wikis, help centres, contracts, code, tickets. Its alternatives each fall short on one axis: fine-tuning teaches style and format far more reliably than it teaches facts, and retraining for every document change is not practical.

You have used it every time an assistant answered with numbered sources, a support bot quoted the help-centre article it relied on, or a "chat with this PDF" tool answered questions about a file it was never trained on. Stripped to the minimum, it is a prompt with retrieved text in it:

RAG is how most products put an LLM in front of real data, and the parts that make it work are ordinary engineering: indexing pipelines, search relevance, access control, caching and evaluation. When a RAG feature gives a confident wrong answer, the cause is usually that the right passage was never retrieved, or that a passage the user was not allowed to see was. Both are fixed in the retrieval layer, not by changing the model or rewording the prompt.

A RAG system is two pipelines that share an index. The indexing pipeline runs offline whenever documents change: load them, strip boilerplate, split them into chunks, embed each chunk, and store the vector together with the text, a keyword index entry and metadata such as tenant, permissions, source URL and date. The query pipeline runs on every question: turn the question into a standalone search query, retrieve candidates the user is allowed to see, rerank them, assemble a prompt with numbered sources, and generate an answer that cites them.

The query rewrite step matters more than it looks. In a conversation, the user's third message is often "and what about yearly ones?", which matches nothing on its own. A cheap model call that rewrites it into "refund policy for annual plans" using the chat history turns a guaranteed miss into a good search.

Chunk size is a trade-off between precision and context. Small chunks match a specific question sharply but lose the sentence that explains them; large chunks keep context but dilute the embedding with unrelated text and use up the prompt budget. Most systems land between a few hundred and roughly a thousand tokens, split along the document's own structure where it has one.

Vector search finds passages that mean the same thing in different words, which is exactly what keyword search misses. It is weak where keyword search is strong: error codes, product SKUs, people's names, version numbers and rare jargon, which an embedding blurs into "something similar". Running both and merging the lists covers both failure modes. The usual merge is RRF (reciprocal rank fusion, 2009): each chunk scores the sum of 1 / (k + rank) over the lists it appears in, with k around 60, so agreement between the two searches matters more than either raw score, and scores on different scales never have to be compared.

Both searches score the query and a chunk separately, which is what makes them fast. A reranker (a cross-encoder) reads the query and one chunk together and scores how well that chunk answers that question, which is much more accurate and much slower, so it only sees the few dozen fused candidates and keeps the best handful for the prompt.

Number the sources and require citations by number, so an answer can be checked against what was retrieved and the interface can link each claim to its document. Tell the model what to do when the sources are insufficient; without that, it fills the gap from memory, which is the hallucination RAG was meant to prevent. And put retrieved text in a clearly delimited data section, never in the system instructions: a document can contain text that reads like an instruction.

Every bad answer has one of two causes. A retrieval miss: the passage with the answer was not in the prompt. A generation miss: it was, and the model ignored, misread or contradicted it. They have different fixes, so the first debugging step is always to look at what was retrieved.

Models also use their context unevenly. Liu and colleagues (2023) found that models were best at using information at the beginning or end of a long prompt and worst in the middle, so stuffing twenty chunks in "to be safe" can lower accuracy while raising cost. A few well-ranked chunks usually beat many mediocre ones.