{"canonical_url":"https://engineer.company/glossary/retrieval-augmented-generation/","content_markdown":"# Retrieval-augmented generation\n\n## What it is\n\nRetrieval-augmented generation is a way of answering a question with a language\nmodel in two steps. First the application searches a collection of documents for\nthe passages most likely to hold the answer. Then it puts those passages into\nthe model's input next to the question, and the model writes its answer from\nthem rather than from what it absorbed during training. The name, usually\nshortened to RAG, comes from a 2020 research paper on knowledge-intensive\nlanguage tasks such as answering open questions.\n\nThat paper described two kinds of memory. A pre-trained model keeps what it\nlearned in its parameters, which the authors called parametric memory; a\nseparate index of documents, searched by a trained retriever, is non-parametric\nmemory. Their system joined a pre-trained sequence-to-sequence model to a dense\nvector index of Wikipedia, and they compared two ways of using what came back:\none conditions the whole answer on the same retrieved passages, the other may\ndraw on different passages for each token it writes. On three open-domain\nquestion-answering benchmarks it set new state-of-the-art results, and the\nauthors found its generated text more specific, more varied and more factual\nthan that of a comparable model without retrieval.\n\nAn application can build the same pattern from parts it already has: a search\nindex over its own documents and a large language model it calls over an API.\nThe retrieval half is ordinary search, and both of its families appear here.\nSparse retrieval matches words, in the way full-text search ranks documents with\na scoring function such as BM25. Dense retrieval turns the question and every\npassage into vectors of numbers and returns the passages whose vectors lie\nclosest, which is vector search. A 2020 paper on dense passage retrieval found\nthat a dense retriever trained on a modest number of question and passage pairs\nbeat a strong BM25 baseline by 9 to 19 percentage points on how often the right\npassage appeared among the first twenty results. A system can run both and merge\nthe two lists.\n\n## Why it matters\n\nA language model knows only what was in its training text, and that text ends on\na fixed date and never included your contracts, your price lists or last week's\nminutes. The RAG paper names two limits of knowledge kept in parameters: it is\nhard to show where an answer came from, and hard to update. Retrieval addresses\nboth. The answer can cite the passages it was given, so a reader can open the\nsource and check it, and new knowledge reaches the model the moment a document\nenters the index, with no training run at all.\n\nIt also puts the question of who may see what back where it belongs. Retrieval\nis a search, and a search can run with the rights of the person asking, so the\nmodel is shown only passages that person could have opened themselves. A model\ntrained on the same documents has no such boundary: everything it learned is\navailable to everyone who uses it.\n\nThe cost is that the answer can be no better than what the search returns. A\nmodel given the wrong passages writes a fluent answer from them, and a model\ngiven nothing useful may answer from memory regardless. Most of the work in a\nretrieval-augmented system therefore goes into the data and the search, not into\nthe model.\n\n## What it looks like in practice\n\nThe trace below follows one question through a retrieval-augmented system. The\nsearch runs with the asking user's permissions, each passage arrives with the\nfile and page it came from, and the instructions tell the model to answer from\nthe passages and nothing else:\n\n```text\nquestion: Which delivery terms apply to order 4471?\nuser:     sales-3 (role: sales, region: north)\n\nretrieved, filtered to what sales-3 may read:\n  [1] contracts/framework-2025.pdf, page 4\n      \"Goods are delivered to the buyer's warehouse\n       unless the order states otherwise.\"\n  [2] orders/4471.txt, line 12\n      \"Delivery terms: as per framework agreement.\"\n\ninstructions:\n  Answer only from the passages below and cite them as [n].\n  If they do not answer the question, say so.\n\nanswer:\n  Order 4471 refers to the framework agreement [2], under\n  which goods are delivered to the buyer's warehouse [1].\n```\n\nEvery part of that trace is data the system can check. A test can confirm that\nthe right passages came back for a known question, a script can confirm that\nevery `[n]` in the answer refers to a passage that was actually retrieved, and a\nperson reading the answer can open page 4 of the contract.\n\nThe groundwork this needs is the subject of this site's page on\n[company data ready for AI agents](/ai-data/). It describes turning old PDFs,\nscans and mail archives into searchable text with their source kept beside them,\nand having every row keep where it came from — the system, the load, the file\nand the page — so an agent can cite the record behind a figure and a person can\nopen it. The page also puts the rules in the data rather than in the prompt: the\nagent acts with its user's privileges, never an administrator's, and every read\nis logged with the person it was made for.\n\n## Where it goes wrong\n\n**The search misses, and the model answers anyway.** When the right passage is\nnot retrieved, the model still receives something and still writes an answer.\nMeasure retrieval on its own: write down real questions together with the\npassages that answer them, and check how often those passages appear among the\nresults the model is given. Tell the model to say so when the passages do not\nanswer the question, and include questions like that among your test cases.\n\n**Passages that cut the answer in half.** Splitting documents into pieces of a\nfixed length separates a table from its heading and a clause from its exception,\nand either half retrieved alone misleads. Split along the structure the document\nalready has, its sections, pages and table rows, and keep the document's title\nand location with every piece.\n\n**The index ignores who is asking.** An index built by a service that can read\neverything will return passages from documents the person asking may not open,\nand the model will summarise them for that person. Filter by the user's\npermissions inside the search, in the index or the database, before anything\nreaches the model. A rule in the prompt is not access control.\n\n**Instructions planted in a document.** Retrieved text is text somebody else\nwrote, and the model cannot reliably tell a passage from an instruction. NIST's\nprofile for generative AI describes indirect prompt injection as attacks on\napplications built on language models through data likely to be retrieved, and a\n2023 paper demonstrated such attacks against real systems. Treat every retrieved\npassage as untrusted input, let no passage trigger an action without a check in\ncode, and give the tools the model can call no more rights than the user has.\n\n**An index that falls behind.** A document is changed or deleted, and the index\nstill holds the old passages, so the system confidently answers from a contract\nthat no longer applies. Update the index whenever a source changes, remove\npassages when their document is deleted, and show the date of each source next\nto the answer.\n\n**Citations that do not say what the answer claims.** A model can cite passage\n[2] for a sentence passage [2] does not support, and NIST notes that generated\ncitations can themselves be confabulated. Check in code that every citation\npoints at a passage that was retrieved, and have people sample answers against\ntheir cited passages as part of regular testing.\n\n## Further reading\n\n- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://arxiv.org/abs/2005.11401)\n  — the 2020 paper that introduced RAG models and compared its two formulations\n  on question answering and generation.\n- [Dense Passage Retrieval for Open-Domain Question Answering](https://arxiv.org/abs/2004.04906)\n  — a dense retriever trained from question and passage pairs, measured against\n  BM25 on finding the passage that holds the answer.\n- [NIST AI 600-1: Generative Artificial Intelligence Profile](https://doi.org/10.6028/NIST.AI.600-1)\n  — NIST's risk profile for generative AI, covering confabulation, data privacy\n  and indirect prompt injection.\n- [Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection](https://arxiv.org/abs/2302.12173)\n  — how instructions hidden in retrieved data take over an application, with\n  attacks on real systems.\n\n\u003chttps://engineer.company/glossary/retrieval-augmented-generation/\u003e\n","date_modified":"2026-10-29T11:17:41Z","description":"How a language model answers from your own documents instead of from memory, what decides whether the answer is right, and the six ways it fails.","formats":{"gemtext":"https://engineer.company/glossary/retrieval-augmented-generation/index.gmi","gopher":"https://engineer.company/glossary/retrieval-augmented-generation/gophermap","html":"https://engineer.company/glossary/retrieval-augmented-generation/","json":"https://engineer.company/glossary/retrieval-augmented-generation/index.json","markdown":"https://engineer.company/glossary/retrieval-augmented-generation/index.md","pdf":"https://engineer.company/glossary/retrieval-augmented-generation/index.pdf"},"kind":"page","language":"en","title":"What is retrieval-augmented generation?","url":"https://engineer.company/glossary/retrieval-augmented-generation/","version":1}