A model’s knowledge is frozen at its training cutoff and contains nothing private to your organisation. Retrieval-augmented generation is the standard answer: when a question arrives, search a corpus for relevant material, paste what you find into the prompt, and ask the model to answer using it.
It is a genuinely simple idea that became the backbone of applied AI, and it is more durable than several waves of predictions of its death.
The pipeline
A conventional RAG system has two phases.
Ingestion, done ahead of time: documents are split into chunks, each chunk is converted to a vector by an embedding model, and the vectors are stored in a searchable index.
Query time: the user’s question is embedded the same way, the index is searched for the nearest chunks, those chunks are inserted into the prompt along with the question, and the model answers from them. Good implementations also return the source of each chunk so the answer can be checked.
The mechanics of embedding and similarity search are covered in tokens and embeddings.
Why it is harder than it sounds
The architecture diagram is trivial. The failure modes are not, and nearly all of them live in retrieval rather than generation.
- Chunking decides your ceiling. Split too small and you sever the context that made a passage meaningful; too large and the embedding averages several topics into something that matches nothing well. Teams routinely discover that chunking strategy affects answer quality more than model choice does.
- Semantic search is bad at exact matches. Part numbers, proper nouns, specific figures, negations. “Contracts that do not include an arbitration clause” is close in vector space to contracts that do. Hybrid retrieval — vector search combined with keyword search — is now the mainstream answer, and enterprise adoption of it has climbed steeply.
- Retrieval failure is invisible. If the right chunk is not retrieved, the model does not announce the gap. It answers from whatever it got, fluently. This is the most dangerous property of the whole pattern.
- Multi-hop questions break single-shot retrieval. Anything requiring two facts combined, or aggregation across many documents, is poorly served by fetching the top few chunks for one query.
Which is why mature systems add machinery: rewriting the query before searching, retrieving generously then reranking with a more expensive model, and letting an agent issue several searches and refine based on what it finds.
Did long context make RAG obsolete?
This was confidently predicted. With roughly a million tokens now standard across frontier models, why retrieve at all — why not send everything?
Retrieval survived, for reasons that are arithmetic rather than ideological. A million tokens is large compared to a document and trivial compared to a corpus; most real knowledge bases are orders of magnitude bigger. Sending a full window on every query is expensive, often with a long-context surcharge on top. Latency scales with input. And as covered in context windows explained, models use material in the middle of very long inputs less reliably than material at the edges — so a stuffed window is not equivalent to a focused one.
What actually happened is more interesting than either prediction. Long context did not replace retrieval; it changed what retrieval is for. Rather than squeezing the three best chunks into a tight budget, systems now retrieve generously and let the model sort through it. Precision matters less; recall matters more.
The framing is also shifting from pipelines to something more like context architecture — agents querying data through purpose-built interfaces at runtime, rather than a fixed pre-query pipeline. The driver is volume: an agent working a task issues far more retrieval calls than a human asking questions, which breaks infrastructure designed for human-scale traffic.
Does it reduce hallucination?
Yes, substantially — and it is the single highest-leverage mitigation available. But it does not eliminate the problem, and it introduces a new one.
A grounded model can still overstate what a source says, blend retrieved material with training-data recall, or reason incorrectly from correct documents. And when retrieval returns something irrelevant but superficially plausible, grounding actively makes the answer worse than abstaining would have been. Citations help the human verify; they do not make the model correct, and a citation generated without an actual retrieved document can itself be fabricated. See why AI models hallucinate.
When to reach for it
Retrieval is the right tool when the model needs knowledge it does not have — private documents, current information, anything that changes. It is the wrong tool for changing how a model behaves, writes, or formats output. That is a different problem, covered in fine-tuning vs RAG vs prompting.
Last reviewed: September 2026.
