NOTES / RAG AND RETRIEVAL

RAG overview

Retrieval-augmented generation in one picture: index your documents, fetch the relevant pieces at question time, and have the model answer from them.

UPDATED 18 NOV 2026#rag#retrieval#grounding

The problem RAG solves

A model’s weights hold what it saw in training. They are hard to update, cannot cite their sources, and do not include your private documents. The original RAG paper framed this as a limit of purely parametric models: they struggle to access and precisely manipulate knowledge, to give provenance for their answers, and to keep their world knowledge up to date1. Its answer was to combine a pretrained generator with a non-parametric memory, a dense index of documents reached through a neural retriever, so that knowledge can be revised and the accessed passages can be inspected1.

In today’s usage “RAG” usually means something simpler than the paper’s end-to-end training: fetch relevant text and put it in the prompt. The idea is the same.

The pipeline

Diagram: index time chunks, embeds and stores documents; query time retrieves, builds a prompt and the model answers

Diagram: my own.

  1. Index time. Split documents into chunks (Chunking), compute representations (Embeddings and vector search, BM25), and store them with metadata.
  2. Query time. Turn the question into a search (Query rewriting), retrieve candidates, optionally fuse and rerank them (Hybrid search and reranking), and place the best passages in the prompt.
  3. Generation. The model answers from those passages, ideally citing them.

In the paper’s experiments, RAG models produced more specific, diverse and factual language than a parametric-only baseline on generation tasks1. That is a research result on older models; measure the effect on your own data.

Where it fails

Diagram: three failure points: retrieval misses, noisy context, unfaithful answer

Diagram: my own, aligned with the quality dimensions in 2.

The Ragas authors name three quality dimensions: whether the retriever finds relevant and focused passages, whether the model uses them faithfully, and the quality of the generation itself2. They also note that too-long contexts make models less effective at exploiting them, especially for information in the middle2, which links back to long-context trade-offs.

Most RAG problems are retrieval problems. Anthropic’s Contextual Retrieval write-up starts from this view: chunks often lose the context they had inside their document, so retrieval misses them3. Fix retrieval first, then look at the prompt.

When to use it

  • Use RAG for private, changing or large knowledge, when you need citations, or when the answer lives in a few passages.
  • Consider long context instead when the whole corpus fits and questions need global reasoning (Long-context trade-offs).
  • Consider fine-tuning for style and behaviour, not for facts that change (When to fine-tune). These guidelines are mine, not from the sources.

Next: Chunking.

Sources

  1. [1]Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks — Patrick Lewis and co-authors · arXiv, NeurIPS 2020 · 2020PARAPHRASED
  2. [2]Ragas: Automated Evaluation of Retrieval Augmented Generation — Shahul Es, Jithin James, Luis Espinosa-Anke, Steven Schockaert · arXiv · 2023PARAPHRASED
  3. [3]Introducing Contextual Retrieval — Anthropic · 2024PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.