NOTES / RAG AND RETRIEVAL

Query rewriting

Fixing the question before searching: rewriting, decomposing and hypothetical-answer techniques that close the gap between how users ask and how documents are written.

UPDATED 18 NOV 2026#query-rewriting#hyde#retrieval

The gap

Users do not write queries the way documents are written. They use pronouns, shorthand and follow-ups (“what about last year?”), or ask something broad that needs several lookups. The Query Rewriting paper puts it as an unavoidable gap between the input text and the knowledge needed for retrieval1. Their idea is to adapt the query, not only the retriever or the reader.

Diagram: rewrite then retrieve then read, and HyDE: draft answer, embed draft, search

Diagram: my own, covering the two techniques below.

Rewrite, retrieve, read

Ma et al. propose a Rewrite-Retrieve-Read pipeline in place of retrieve-then-read: first prompt a language model to write a search query from the user’s input, then retrieve, then read1. They also train a small language model as a rewriter, using reinforcement learning with feedback from the downstream reader, so the rewriter adapts to a frozen retriever and a black-box LLM. They report consistent improvements on open-domain and multiple-choice QA1.

You can use the simple prompted version without any training.

Hypothetical answers (HyDE)

Questions and answers often look different, so a question’s embedding can sit far from the passages that answer it. HyDE works around this: prompt an instruction-following model to write a hypothetical document that answers the query, embed that, and use the embedding to find real, similar documents. The generated text may contain false details, but the encoder’s compression filters them out and the search is grounded in the real corpus2. The authors report that HyDE significantly outperformed the unsupervised dense retriever Contriever and was comparable to fine-tuned retrievers across tasks and languages2.

Trade-off: an extra model call per query and the risk of steering toward a wrong guess. It helps most when queries are short and documents are long.

Other useful rewrites

These are common techniques from practice rather than from the two sources:

  • Resolve context. In a conversation, rewrite a follow-up into a standalone question using the chat history.
  • Decompose. Split a multi-part question into sub-queries, retrieve for each, and merge. This is the bridge to Agentic RAG.
  • Expand. Generate several paraphrases or keyword variants, search with all, and fuse the lists (RRF).
  • Extract filters. Pull dates, entities or document types from the question and apply them as metadata filters.

Guardrails

  • Keep the original query too. Search with both the rewritten and the original text so a bad rewrite cannot hide the obvious match.
  • Log rewrites. Inspect them in your traces; they are a frequent hidden source of errors.
  • Test the gain. Compare with and without rewriting on a real eval set. Extra latency must pay for itself.
  • Mind injection. Rewriting takes user text through a model; see Prompt injection.

Next: Agentic RAG.

Sources

  1. [1]Query Rewriting for Retrieval-Augmented Large Language Models — Xinbei Ma, Yeyun Gong, Pengcheng He, Hai Zhao, Nan Duan · arXiv, EMNLP 2023 · 2023PARAPHRASED
  2. [2]Precise Zero-Shot Dense Retrieval without Relevance Labels — Luyu Gao, Xueguang Ma, Jimmy Lin, Jamie Callan · arXiv · 2022PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.