Retrieval augmented generation (RAG): how AI systems back up answers with current sources

What is retrieval augmented generation (RAG)?

Retrieval augmented generation combines two building blocks. A search system finds suitable content, and a large language model turns it into the answer. The spelling retrieval-augmented generation is also common. Google calls the technique grounding.

Without retrieval, an LLM only knows what is in its training data. According to AWS, RAG lets it draw on an external knowledge source without being retrained. This is how newer or internal information gets into the answer.

The term goes back to a paper by Patrick Lewis and colleagues from 2020, which was accepted for the NeurIPS conference.

How RAG works: retrieval, augmentation, generation

For every question, a RAG system goes through three steps:

1. Retrieval: the system looks for content that matches the question, for example in a search index or its own database.
2. Augmentation: the passages it finds are added to the question and passed to the language model together.
3. Generation: the model writes the answer on that basis. Many systems also name or link to their sources.

If a customer asks a manufacturer’s chatbot about the warranty period, for example, the system finds the relevant paragraph in the warranty terms. It passes this paragraph to the model together with the question. The model receives something like this text and answers on that basis:

Question: How long is the warranty on model X?

Context from “Warranty terms, section 3”:
Model X comes with a 24-month warranty from the date of purchase.

Instruction: Answer only on the basis of the context and name the source.

The building blocks of a RAG system

• Knowledge base: the documents, databases or website content the system draws on.
• Retriever: the search component that finds passages matching a question.
• Prompt assembly: the logic that combines the question, passages and instructions.
• Language model: the model that writes the answer.

Chunks, embeddings and vector search

To find suitable passages quickly, many RAG systems split documents into short chunks in advance, usually no more than a few hundred tokens long, according to Anthropic. Each chunk is converted into a numeric vector, known as an embedding, which represents its meaning. The question goes through the same step, and a vector search finds the chunks with a similar meaning. A keyword search such as BM25 can also find exact words and phrases.

Combining them pays off. Anthropic reports from its own tests that chunks with added context, together with a keyword search, reduced the retrieval failure rate by 49%, and by 67% with an additional reranking step. These are the vendor’s figures for its test data.

Benefits of RAG

• Current knowledge: new content goes into the knowledge base, and the model can use it for the next question. This does not require new training.
• Your own data: a company’s manuals, product data or policies flow into the answer even though the model never learned them.
• Verifiable answers: the system knows which passages it used and can name them as sources.
• Fewer invented details: the model relies on the texts it is given. This does not rule out errors, as the section on limits shows.
• Controllable access rights: retrieval can be limited to documents the person asking may see. The system has to provide for this check explicitly.

How AI search systems use RAG

AI search systems retrieve web pages for a question and use them to write an answer with sources. In its guide to generative AI search, Google names RAG as a technique for improving the quality, accuracy and freshness of AI responses. According to Google, its core ranking systems fetch relevant, current pages from the search index. The AI systems review the information on those pages and show links to the pages that support the answer. This applies to AI Overviews, for example.

OpenAI describes a similar process for web search in its API. The model searches the web, can open pages and shows the addresses it used as sources. The providers do not disclose the detailed rules they use to select pages.

Not every AI answer is based on retrieval. Without a search, a model answers from what it learned in training. Training and retrieval are two separate ways for content to reach an answer. OpenAI, for example, uses different crawlers for them. See AI crawlers for more.

How RAG differs from related terms

RAG describes the technique on the AI system’s side. The other terms refer to the model, a sub-step of the search or the work on your website:

RAG or fine-tuning?

With fine-tuning, you train a base model with examples of expected inputs and outputs, as OpenAI describes it. According to OpenAI’s documentation, its fine-tuning platform is no longer open to new users. RAG leaves the model unchanged and gives it suitable content for each question. The two approaches solve different tasks:

The two methods can be combined. For small knowledge bases there is a third route. According to Anthropic, you can put a knowledge base under 200,000 tokens into the prompt in full and skip retrieval.

What RAG means for website content

A RAG system can only use content that it finds and is allowed to fetch. This leads to five requirements for your pages:

• Accessible: the page must be reachable for the provider’s search crawlers. For Google, it must be indexed and eligible to be shown with a snippet.
• Findable: Google uses its core ranking systems for retrieval. Search engine optimisation therefore remains the foundation.
• Understandable on its own: name the product, time period or point of reference in the paragraph itself. According to Anthropic, chunks taken out of a document can easily lose their context otherwise.
• Unambiguous: prices, dates and key figures should be up to date and consistent across your pages.
• Original: Google recommends content with your own expertise or experience that goes beyond common knowledge.

You do not need to break up your content for this. Google states that content for generative search does not have to be split into tiny pieces or written specifically for AI. Clear headings and paragraphs are enough for the structure. The specific measures are covered in the article on generative engine optimisation.

When RAG suits your project

A RAG system of your own is worthwhile if these points apply:

• The answers depend on documents that change or are not public.
• Users should be able to check where a statement comes from.
• The knowledge base is too large to give to the model in full with every question.

Typical uses are customer service chatbots, internal knowledge search and product advice in online shops. Text retrieval alone is less suitable for questions that require counting or calculating, such as quarterly revenue. A database query is the right retrieval route for those.

Limits of RAG

RAG lowers the risk of wrong answers but does not rule it out. AWS calls the method one approach to some of the problems of language models. If the system finds the wrong passage or the source is out of date, the answer repeats the error. The model can also misrepresent a correct source. See AI hallucinations for more.

Common sources of error when building a system

• Chunks without context: a paragraph without the product name is not found or is matched to the wrong product.
• Vector search alone: a keyword search often finds item numbers and exact names more reliably.
• Old and new versions: if both sit in the knowledge base, the system can retrieve the outdated one.
• No access check: retrieval returns documents the person asking is not allowed to see.
• No testing: changes go live without their effect being measured.

Evaluating RAG systems

Check retrieval and answers separately. The paper on Ragas names three areas. Does retrieval find suitable passages? Does the model reproduce them faithfully? How good is the answer overall? Create a list of typical questions for this and repeat the test after every change.