NOTES / RAG AND RETRIEVAL

Agentic RAG

Letting the model run the retrieval: decide whether to search, what to search for, and when it has enough, instead of one fixed retrieve-then-answer pass.

UPDATED 18 NOV 2026#agentic-rag#self-rag#agents#retrieval

From a fixed pipeline to a loop

Basic RAG retrieves once, for every question, and answers. That is wasteful when the question needs no lookup and insufficient when one search is not enough. Agentic RAG puts a model in charge of the retrieval process.

Diagram: a fixed pipeline retrieves once; an agentic loop decides whether it has enough or searches and refines again

Diagram: my own.

A survey of the area describes Agentic RAG as embedding autonomous agents into the RAG pipeline, using agentic design patterns, namely reflection, planning, tool use and multi-agent collaboration, to manage retrieval strategies dynamically and refine context iteratively1. The patterns themselves are covered in Agentic design patterns and Reflection.

Retrieval as a tool

The simplest agentic RAG is to expose search as a tool and let the model call it (Tool calling). The model can then:

  • skip retrieval when it already knows or needs none,
  • write its own queries, and reformulate after a poor result,
  • break a question into sub-questions and search each,
  • search several sources, such as a vector index, a keyword index and a database,
  • stop when the evidence suffices (Stopping conditions).

Anthropic’s context engineering post describes the same direction, with agents keeping lightweight references and loading data into context on demand through tools instead of preloading everything3. My own observation: for well-structured sources such as codebases, direct tools like file search can sometimes replace a vector index altogether. See Context window as a budget.

Self-RAG: a trained version

Self-RAG shows one way to build this into the model. Instead of always retrieving a fixed number of passages, which can hurt versatility when retrieval is unneeded or irrelevant, it trains a model to retrieve on demand and to critique itself using special reflection tokens that judge retrieved passages and its own output2. The authors report that 7B and 13B Self-RAG models outperformed ChatGPT and retrieval-augmented Llama2-chat on several tasks, with better factuality and citation accuracy in long-form generation2. In practice most teams get a similar effect by prompting a general model, not training one.

Costs and risks

  • More calls, more latency and cost. Each loop iteration is a model call plus a search.
  • Loops that do not end. Cap iterations and set budgets (Levels of autonomy).
  • Compounding errors. A bad early query can send the whole chain wrong. Log every query and result (Tracing and monitoring).
  • Harder to evaluate. There is no single retrieval to score; you judge trajectories as well as answers.
  • Injection surface. Retrieved text is untrusted input that now steers further actions (Prompt injection).

When to choose it

Use a fixed pipeline when questions are uniform and one search usually answers them. Move to agentic RAG when questions are multi-hop, sources are heterogeneous, or you need the system to notice and recover from a failed search. This is my guidance, not a finding from the sources, and the Choosing a pattern note applies: start with the simplest thing that works.

Next: Measuring retrieval.

Sources

  1. [1]Agentic Retrieval-Augmented Generation: A Survey on Agentic RAG — Aditi Singh, Abul Ehtesham, Saket Kumar, Tala Talaei Khoei, Athanasios V. Vasilakos · arXiv · 2025PARAPHRASED
  2. [2]Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection — Akari Asai, Zeqiu Wu, Yizhong Wang, Avirup Sil, Hannaneh Hajishirzi · arXiv · 2023PARAPHRASED
  3. [3]Effective context engineering for AI agents — Anthropic Engineering · 2025PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.