NOTES / RAG AND RETRIEVAL

Hybrid search and reranking

Combine keyword and vector retrieval, fuse the rankings with reciprocal rank fusion, then let a slower reranker pick the few passages that go in the prompt.

UPDATED 18 NOV 2026#hybrid-search#reranking#rrf#retrieval

Why combine

Keyword and vector search fail differently (BM25, Embeddings). The DPR authors found that the dense representation complements the sparse one, and that combining them improved retrieval further4. Anthropic’s standard RAG recipe likewise combines embeddings and BM25 and merges the results3.

Diagram: the query runs through BM25 and vector search in parallel, results are fused, then reranked to a top k

Diagram: my own.

Fusing the lists

The two retrievers return scores on different scales, so adding raw scores is awkward. Reciprocal rank fusion (RRF) sidesteps this by using only ranks: each document’s fused score is the sum, over the lists it appears in, of one divided by a constant plus its rank. Cormack, Clarke and Büttcher reported that this simple unsupervised method consistently beat any individual system it combined, and beat the Condorcet Fuse method1.

Diagram: two ranked lists fused by summing reciprocal ranks, so items high in both rise

Diagram: my own.

Strengths: no training, no score calibration, easy to explain. Limits: it ignores score magnitudes, and the constant is a tunable choice. Weighted fusion or a learned combination are alternatives if you have labelled data.

Reranking

A first-stage retriever must be cheap, because it scans everything. A reranker scores a small candidate set more carefully, typically by reading the query and passage together. Nogueira and Cho describe retrieval as a pipeline where the second stage re-ranks the top candidates with a more computationally intensive method; their BERT reranker improved on the previous state of the art by 27 percent relative on MS MARCO passage ranking in MRR@102. BEIR likewise found re-ranking and late-interaction models achieved the best zero-shot results on average, but at high computational cost5.

Diagram: a funnel from about 100 candidates to a reranker to about 5 passages

Diagram: my own. The counts are typical orders of magnitude, not recommendations.

Anthropic measured the stack: contextual embeddings plus contextual BM25 cut top-20 retrieval failures by 49 percent, and adding reranking brought the reduction to 67 percent3. These are vendor-reported numbers on their datasets; the pattern, each stage helping, is the useful part.

Practical guidance

  • Start hybrid. It is a small step beyond vector-only and protects against exact-term misses.
  • Retrieve wide, rerank narrow. Pull enough candidates that the right one is likely present, then rerank to a handful.
  • Budget for latency and cost. A reranker adds a call per candidate; cap the candidate count and measure.
  • Evaluate each stage. Check recall of the candidate set, then precision after reranking (Measuring retrieval).
  • Use filters and metadata before ranking when you can; narrowing the pool is cheaper than ranking all of it.

Next: Query rewriting.

Sources

  1. [1]Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods — Gordon V. Cormack, Charles L. A. Clarke, Stefan Büttcher · ACM SIGIR 2009 · 2009PARAPHRASED
  2. [2]Passage Re-ranking with BERT — Rodrigo Nogueira, Kyunghyun Cho · arXiv · 2019PARAPHRASED
  3. [3]Introducing Contextual Retrieval — Anthropic · 2024PARAPHRASED
  4. [4]Dense Passage Retrieval for Open-Domain Question Answering — Vladimir Karpukhin and co-authors · arXiv, EMNLP 2020 · 2020PARAPHRASED
  5. [5]BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Nandan Thakur and co-authors · arXiv, NeurIPS 2021 · 2021PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.