NOTES / PROMPTING AND CONTEXT ENGINEERING
Long-context trade-offs
When to put everything in a huge window and when to retrieve instead: quality, latency and cost all move with length, so decide by measuring.
The temptation
Windows now reach hundreds of thousands or millions of tokens. It is tempting to paste in the whole corpus and ask. Sometimes that is exactly right. Often it is slower, pricier and less accurate than a smaller, better-chosen context.
What goes wrong at length
- Position effects. In multi-document question answering, Liu et al. found performance fell when the relevant passage was in the middle of the input, compared with the start or end, and that accuracy dropped as context lengthened, even for long-context models1.
- Non-uniform degradation. Chroma reported that reliability declines as inputs grow, in uneven and sometimes surprising ways, and that similar-looking distractors hurt more2.
- Cost and latency. More input tokens cost more and take longer to process. See Latency and cost and KV cache.
Models have improved since these studies, so treat them as warnings to measure, not fixed numbers.
Stuff or retrieve?
Diagram: my own.
| Situation | Lean toward |
|---|---|
| One document or a small, stable set; tasks need global reasoning across it | Long context |
| Large or changing corpus; questions touch a few passages | Retrieval (RAG overview) |
| Same big prefix reused across many calls | Long context plus prompt caching |
| Need citations to exact sources | Retrieval, or long context with quoting |
| Agent exploring a codebase or files | Just-in-time lookup rather than preloading4 |
That table is my synthesis, not a rule from the sources.
If you do use long context
Anthropic’s guide gives specific advice for inputs of 20k tokens and above. Put long documents near the top of the prompt, above the question, and wrap each document in tags with metadata. I also recommend placing the question after the documents and, for long inputs, asking the model to quote the relevant passages before answering. I am relying on summaries of the guide for those two points rather than its full text, so check the current page3.
Diagram: my own, following the advice in 3.
Also:
- Trim what is not needed. Even with a big window, shorter inputs are cheaper and often better4.
- Evaluate at your real lengths. Build an eval with the relevant fact placed at different positions, and compare stuffed against retrieved.
- Cache the stable prefix. Order the prompt so unchanging content comes first.
Next: Compaction and summarisation.
Sources
- [1]Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu and co-authors · arXiv, TACL · 2023PARAPHRASED
- [2]Context Rot: How Increasing Input Tokens Impacts LLM Performance — Kelly Hong, Anton Troynikov, Jeff Huber · Chroma · 2025PARAPHRASED
- [3]Prompting best practices — Anthropic, Claude Platform docsPARAPHRASED
- [4]Effective context engineering for AI agents — Anthropic Engineering · 2025PARAPHRASED