NOTES / PROMPTING AND CONTEXT ENGINEERING

The context window as a budget

Everything the model sees competes for one finite window and one finite attention budget. Treat tokens as a scarce resource and keep only high-signal ones.

UPDATED 18 NOV 2026#context-window#attention#context-rot#budget

One window, many claimants

The context window is everything the model can consider in a call: system prompt, tool definitions, conversation history, retrieved documents, tool results, and room for the answer. All of it shares one token limit.

Diagram: a bar split into system prompt, tools, history, retrieved content and reserved output

Diagram: my own. The proportions are illustrative.

In an agent, history and tool results grow every turn, so the budget is spent faster than people expect. Count tokens with the target model’s own tokenizer, since counts differ between models (Tokens and tokenisation).

A second budget: attention

Fitting is not the only constraint. Anthropic frames context as a finite resource with diminishing returns, and says models have an “attention budget” that gets stretched as context grows, because every token attends to every other token1. Their stated goal is the smallest set of high-signal tokens that makes the desired outcome likely1.

Research backs the intuition. Chroma tested many models and reported that models do not use their context uniformly: performance becomes increasingly unreliable as input length grows, even on simple tasks, and distractors make it worse2. Liu et al. found accuracy depends on where the relevant information sits, best at the start or end and worst in the middle3.

Diagram: an illustrative curve of reliability falling as input grows longer

Diagram: my own and illustrative only. Real curves differ by model and task; measure yours.

Spending the budget well

My working rules, built on the sources above:

  1. Audit what is in the window. Log the full prompt for a sample of calls and see what actually occupies it. It is often mostly stale tool output.
  2. Keep tools few and clear. Anthropic notes that overlapping tools make it harder for the model, and for humans, to decide which to use1. See Tool design.
  3. Load just in time. Instead of pre-loading everything, keep lightweight references such as file paths or queries and fetch details when needed; Anthropic describes this as retrieving context dynamically at runtime1.
  4. Return small tool results. Trim, paginate or summarise what tools hand back.
  5. Prune and compact as you go. See Compaction and summarisation.
  6. Isolate with sub-agents when a side task would flood the main context. See Multi-agent systems.
  7. Reserve room for output. Reasoning and long answers need tokens too.

Next: Long-context trade-offs.

Sources

  1. [1]Effective context engineering for AI agents — Anthropic Engineering · 2025PARAPHRASED
  2. [2]Context Rot: How Increasing Input Tokens Impacts LLM Performance — Kelly Hong, Anton Troynikov, Jeff Huber · Chroma · 2025PARAPHRASED
  3. [3]Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu and co-authors · arXiv, TACL · 2023PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.