NOTES / INFERENCE AND SERVING

The KV cache

What the key-value cache stores, why it dominates serving memory, and the techniques that shrink or manage it: paging, grouped-query attention and prefix reuse.

UPDATED 18 NOV 2026#inference#kv-cache#pagedattention#memory

What it stores

Self-attention compares each new token against all earlier tokens. The keys and values computed for earlier tokens do not change, so the model keeps them instead of recomputing them at every step. That store is the KV cache.

Diagram: without a cache each step recomputes the prefix; with a cache the stored keys and values are reused and only the new token's are computed and appended

Diagram: my own.

Prefill fills the cache for the prompt. Decode reads it at every step and appends one token’s worth. It is a trade of memory for compute, and the memory is paid per request.

Why it matters

A common formula for its size is layers times KV heads times 2 (keys and values) times sequence length times head dimension times bytes per element3. It grows linearly with context length and with the number of concurrent requests. On a GPU the model’s weights are fixed, and what is left over goes to the KV caches of all active requests, so the cache limits how many requests you can batch.

The PagedAttention paper makes this point: each request’s KV cache is large and grows and shrinks dynamically, so inefficient management wastes memory through fragmentation and duplication and limits batch size1.

Paging the cache

PagedAttention borrows virtual memory from operating systems. It splits each request’s KV cache into fixed-size blocks that need not be contiguous, so blocks are allocated as the sequence grows1. Blocks can also be shared across sequences, including across requests with a common prefix1. The vLLM engine built on it reports near-zero cache waste and 2 to 4 times the throughput of the prior systems it compared against, at the same latency1. Treat the multipliers as the authors’ benchmark results, not a promise for your workload.

Diagram: a contiguous reservation leaves unused slots; paged blocks are allocated on demand and interleave across requests

Diagram: my own.

Making the cache smaller

Fewer K/V heads. In multi-head attention every query head has its own keys and values. Multi-query attention shares one K/V head across all query heads, and grouped-query attention shares one per group, so the cache scales with the number of distinct K/V heads rather than query heads2. This is a model design choice made at training time, so you benefit by picking models that use it.

Diagram: multi-head attention stores 8 K/V sets, grouped-query 2 and multi-query 1

Diagram: my own.

Lower precision. Some engines can store the cache in fewer bits. This reduces memory at some risk to accuracy (my own note; check your engine’s docs and test).

Shorter contexts. The cheapest cache is the one you do not fill. Trim what you send (Context window as a budget, Long-context trade-offs).

Reusing the cache across requests

If two requests start with the same tokens, their KV entries for that prefix are identical, so a server can keep and reuse them. SGLang’s RadixAttention is one design for this: it reuses the KV cache across generation calls that share prefixes4. Hosted APIs expose the same idea as prompt caching. The saving only applies to an exact shared prefix, which is why prompt layout matters.

Practical takeaways

  1. Budget memory as weights plus per-request cache, not weights alone.
  2. When you see out-of-memory errors or small batches on long prompts, suspect the cache.
  3. Prefer models with grouped-query attention if you self-host.
  4. Put stable content first so prefixes can be reused.
  5. Remember that long agent runs keep growing their cache and their bill (Latency and cost).

Sources

  1. [1]Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al. · 2023PARAPHRASED
  2. [2]Reducing Transformer Key-Value Cache Size with Cross-Layer Attention — Brandon et al. · 2024BACKGROUND SECTION ON MQA AND GQA
  3. [3]Multi-Query and Grouped-Query Attention: Shrinking the KV Cache — General Compute · 2026CACHE-SIZE FORMULA, AS SUMMARISED BY A THIRD PARTY
  4. [4]SGLang: Efficient Execution of Structured Language Model Programs — Zheng et al. · 2024PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.