NOTES / RAG AND RETRIEVAL

Embeddings and vector search

Turning text into vectors so meaning becomes distance, and searching millions of them fast with approximate nearest-neighbour indexes such as HNSW.

UPDATED 18 NOV 2026#embeddings#vector-search#hnsw#dense-retrieval

What an embedding is

An embedding is a list of numbers, a vector, that represents a piece of text so that texts with similar meaning end up close together. Retrieval then becomes geometry: embed the query, find the nearest document vectors.

Diagram: texts as points in a space, with the query's nearest neighbours circled

Diagram: my own, in two dimensions for illustration. Real embeddings have hundreds or thousands.

Where this came from

Plain BERT needs both sentences fed in together to compare them, which makes searching a large set very slow. Sentence-BERT used siamese networks to produce fixed-size sentence vectors that can be compared with cosine similarity. The authors report that finding the most similar pair among 10,000 sentences drops from about 65 hours with BERT to about 5 seconds, with similar accuracy1.

For passage retrieval, Dense Passage Retrieval trains a dual encoder: one encoder for questions and one for passages, learned from a modest number of question and passage pairs. It beat a strong Lucene BM25 baseline by 9 to 19 percentage points in top-20 passage retrieval accuracy on open-domain QA datasets2. The authors also observed that dense and sparse representations are complementary, so combining them can improve results further2. That is the basis of hybrid search.

Using embeddings in practice

  • Use the same model for queries and documents (or a model designed as a query and document pair), and record its name and version with the index. Changing it means re-embedding everything.
  • Normalise and pick the metric the model expects, often cosine similarity or inner product.
  • Mind the input limit. Embedding models have maximum input lengths; this ties back to chunking.
  • Domain matters. General models can miss specialised vocabulary. BEIR, a benchmark of 18 datasets from diverse retrieval tasks used for zero-shot evaluation, found that performing well consistently across all of them is hard4. Test candidates on your data, not only on leaderboards.
  • Embeddings are not a database of facts. They capture topical similarity, not truth or exact identifiers.

Searching many vectors quickly

Comparing a query with every vector is exact but scales linearly. Production systems use approximate nearest-neighbour (ANN) indexes that trade a small amount of recall for large speed gains. A common choice is HNSW, which builds a multi-layer structure of proximity graphs over nested subsets of the points. Search starts in a sparse upper layer and moves down, which the authors show gives roughly logarithmic scaling3.

Diagram: three layers of a graph, sparse at the top and dense at the bottom, with search descending

Diagram: my own simplification of the idea in 3.

Knobs you will meet, in general terms: how many neighbours each node links to, how wide a candidate list to explore while building and while searching. Wider search raises recall and latency together. Whatever the database, measure recall against exact search on your own vectors so you know what the approximation costs.

Filters and metadata

Real queries often need filters such as tenant, date range or document type. Check how your index applies filters, since pre-filtering and post-filtering can change both results and speed. This is practical advice from experience, not a claim from the sources.

Next: BM25 and sparse retrieval.

Sources

  1. [1]Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — Nils Reimers, Iryna Gurevych · arXiv, EMNLP 2019 · 2019PARAPHRASED
  2. [2]Dense Passage Retrieval for Open-Domain Question Answering — Vladimir Karpukhin and co-authors · arXiv, EMNLP 2020 · 2020PARAPHRASED
  3. [3]Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs — Yu. A. Malkov, D. A. Yashunin · arXiv, IEEE TPAMI · 2016PARAPHRASED
  4. [4]BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models — Nandan Thakur, Nils Reimers, Andreas Rücklé, Abhishek Srivastava, Iryna Gurevych · arXiv, NeurIPS 2021 · 2021PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.