NOTES / RAG AND RETRIEVAL

Chunking

Splitting documents into retrievable pieces: how chunk size and boundaries change what can be found, and how to restore the context chunks lose.

UPDATED 18 NOV 2026#rag#chunking#indexing

Why chunk at all

Retrievers match a query against units of text. A whole document is usually too big and too mixed to be a good match, and cannot all go in the prompt. So you split documents into chunks, index each one, and retrieve the best few. The split decides what the retriever can ever return.

The size trade-off

Diagram: small chunks match precisely but may lose context; large chunks carry context but add noise

Diagram: my own.

Small chunks make precise matches and keep prompts lean, but a sentence cut from its surroundings may be ambiguous. Large chunks keep context but dilute the match and spend more tokens on text that is not relevant. There is no universal best size; it depends on your documents and questions.

Chroma’s evaluation found that the choice of chunking strategy has a significant impact on retrieval accuracy and efficiency1. They argue that common benchmarks judge retrieval at the level of whole documents, which hides the effect of chunking, so they evaluate at the token level using precision, recall and intersection-over-union against the excerpts that answer a question1. Anthropic likewise lists chunk size, boundaries and overlap as choices that affect retrieval performance2.

Strategies

Diagram: fixed size, recursive, semantic and parent and child chunking

Diagram: my own.

  • Fixed size with optional overlap is the simplest baseline.
  • Recursive splitting tries natural separators first, such as paragraphs, then sentences.
  • Semantic splitting places boundaries where meaning shifts. Chroma tested embedding-based and LLM-driven chunkers of this kind1. They cost more at index time.
  • Parent and child retrieves by small chunks but returns the larger section around the match. This is a widely used pattern; I am describing it from general practice rather than from these sources.
  • Structure-aware splitting uses headings, tables and code blocks, which usually carry meaning. Respect them.

Overlap between neighbouring chunks reduces the chance that an answer is cut in half, at the cost of duplication. Keep it modest and tune it.

Restoring lost context

A chunk like “revenue grew 3% over the previous quarter” does not say which company or quarter. Anthropic’s Contextual Retrieval addresses this by having a model write a short, chunk-specific explanation, usually 50 to 100 tokens, and prepending it to the chunk before embedding it and before building the BM25 index2. In their tests, contextual embeddings cut the top-20 retrieval failure rate by 35 percent (5.7 to 3.7 percent), and combining contextual embeddings with contextual BM25 cut it by 49 percent (5.7 to 2.9 percent)2. These are the vendor’s numbers on their own datasets.

Cheaper first steps: prepend the document title and section heading to every chunk, and store metadata such as source, date and section for filtering. See Prompt caching for reducing the cost of generating contexts.

Working method

  1. Start with recursive splitting and a moderate size.
  2. Build a small eval set of real questions with the passages that answer them (Measuring retrieval).
  3. Change one thing at a time: size, overlap, strategy, context prefix. Keep what moves the metric.
  4. Re-run after you change embedding models, since the best chunking can shift.

Next: Embeddings and vector search.

Sources

  1. [1]Evaluating Chunking Strategies for Retrieval — Brandon Smith, Anton Troynikov · Chroma · 2024PARAPHRASED
  2. [2]Introducing Contextual Retrieval — Anthropic · 2024PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.