Sampling and decoding
How logits become a chosen token: temperature, top-k and top-p, why pure sampling degenerates, and what to do when providers remove the knobs.
From scores to a token
At each step the model outputs a score (a logit) for every token in its vocabulary. A decoding strategy turns those scores into one chosen token. Run it in a loop and you get text.
Diagram: my own.
The simplest strategy is greedy: always take the highest-scoring token. It is deterministic, but for open-ended text it tends to be repetitive. Sampling from the full distribution fixes repetition but brings a different problem.
Why pure sampling fails
Holtzman and colleagues studied text degeneration, where output is bland, incoherent or stuck in repetitive loops1. They found that decoding strategy alone can dramatically change text quality, even from exactly the same model1. Their diagnosis for pure sampling is the “unreliable tail”: many individually unlikely tokens that together hold enough probability to be picked often, derailing the text1.
Their remedy, nucleus (top-p) sampling, samples only from the smallest set of tokens whose cumulative probability reaches p, which the authors call the nucleus. That set expands and contracts as the distribution changes1. They report it as the best of the strategies compared for long-form text that is both high quality and diverse1.
Diagram: my own. Probabilities come from an example set of logits.
Top-k, by contrast, keeps a fixed number of top tokens regardless of the distribution’s shape. When the model is very sure, k may include junk; when it is unsure, k may cut good options1.
Temperature
Temperature rescales the logits before softmax. Low temperature sharpens the distribution toward the top token; high temperature flattens it. Temperature near zero approaches greedy decoding.
Diagram: my own, computed from the logits 3.0, 2.2, 1.6, 0.8, 0.2 at temperatures 0.5, 1 and 2.
Holtzman et al. also note that lowering temperature reduces the tail problem but costs diversity and brings repetition1, which is why truncation methods like top-p were proposed alongside it.
Practical settings
My working defaults, from general experience rather than the sources:
- Extraction, classification, code, tool arguments: low temperature, for consistency.
- Brainstorming, varied phrasing: moderate to higher temperature.
- Change one knob at a time. Temperature and top-p both shape the same distribution.
- Do not rely on determinism. Even at zero temperature, outputs can vary across hardware and batching, so treat equality tests with care and use evals rather than exact matching.
The knobs may not exist
Providers are retiring these controls on newer models. One compatibility reference reports that on Claude Opus 4.7 and later, a non-default temperature, top_p or top_k returns a 400 error, and that Google has deprecated them on newer Gemini models2. This is a secondary source and moves fast, so check the current API docs for your model. When sampling controls are gone, steer behaviour through the prompt, structured outputs, and reasoning effort instead (see Reasoning models).
Sources
- [1]The Curious Case of Neural Text Degeneration — Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi · arXiv, ICLR 2020 · 2019PARAPHRASED
- [2]LLM API Parameter Compatibility Reference — Hidekazu Konishi · hidekazu-konishi.com · 2026PARAPHRASED