NOTES / INFERENCE AND SERVING

Batching

Why serving many requests together raises throughput, how continuous batching refills the batch at every step, and how the trade-off with latency shows up.

UPDATED 18 NOV 2026#inference#batching#continuous-batching#throughput

Why batch

At small batch sizes, decoding is limited by reading the model’s weights from memory for every token. If several requests are processed in the same step, the weights are read once and used for all of them, so each extra request is cheap. That is why servers group requests: it raises tokens per second per GPU and lowers cost per token. The Orca authors note the reason for the matrix operations: reusing loaded parameters across requests reduces memory reads, whereas attention has no parameters to reuse, so batching it gives no such benefit2.

The cost is latency for individual requests if the batch is too large or badly scheduled. See Latency and cost.

Static batching wastes capacity

In the simplest design, the server takes a batch of requests and runs them until all finish. Orca’s paper describes the problem: with request-level scheduling, requests that finish early cannot be returned to the client, and newly arrived requests wait until the whole batch completes1. Because output lengths vary, short requests sit idle while long ones finish.

Continuous batching

Orca proposed iteration-level scheduling: the scheduler runs a single model iteration on the batch, then checks again, so a finished request leaves and a new one can join after only one iteration of waiting1. This is usually called continuous batching. Frameworks use different names for the same idea: continuous batching in vLLM and Anyscale’s write-up, in-flight batching in NVIDIA’s TensorRT-LLM, and iteration-level scheduling in the research literature4.

Diagram: static batching leaves idle slots until the slowest request finishes; continuous batching refills freed slots at every step

Diagram: my own.

Orca reported a 36.9 times throughput improvement over NVIDIA FasterTransformer at the same latency on a GPT-3 175B model1. That is a result from one system, model and workload in 2022; modern engines combine it with other optimisations, so do not expect that exact factor.

Prefill and decode in one batch

Prefill is heavy and parallel, and decode is light and sequential. A long prefill in the batch can stall everyone else’s decode steps and spike their inter-token latency. vLLM’s docs list chunked prefill, which splits long prompts so decode steps keep flowing6. Some stacks go further and run prefill and decode on separate GPUs; vLLM’s documentation index lists disaggregated prefill as a feature, though you should check its current status6.

Memory also limits batch size, because each active request needs its own KV cache. Paging the cache lets more requests fit at once3.

Two different meanings of “batch”

Do not confuse server-side batching with batch APIs. A batch API such as Anthropic’s lets you submit many requests together for asynchronous processing at a lower price, with results within 24 hours5. That is a pricing and scheduling product for non-urgent work. Continuous batching happens inside the server for live traffic.

What to do as an application developer

  • If you use a hosted API, the provider handles batching. Your levers are concurrency, request size and using the batch API for offline jobs.
  • If you self-host, use an engine with continuous batching and tune maximum batch size, maximum tokens in flight and prefill chunking against your latency target (Serving stacks).
  • Measure under load. A model that looks fast with one user can behave very differently at 50 concurrent requests. Track p95 TTFT and inter-token latency while raising concurrency.
  • Protect the tail. Cap prompt and output lengths, and prioritise interactive traffic over bulk jobs.

Sources

  1. [1]Orca: A Distributed Serving System for Transformer-Based Generative Models — Yu et al. · OSDI · 2022PARAPHRASED
  2. [2]Orca paper (full text) — Yu et al. · 2022PARAPHRASED
  3. [3]Efficient Memory Management for Large Language Model Serving with PagedAttention — Kwon et al. · 2023PARAPHRASED
  4. [4]Continuous Batching: How It Works and Why It Multiplies LLM Throughput — MorphNAMING ACROSS FRAMEWORKS, AS SUMMARISED BY A THIRD PARTY
  5. [5]Message Batches API — Anthropic docsPARAPHRASED
  6. [6]Welcome to vLLM — vLLM project docsPARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.