NOTES / LLM FUNDAMENTALS

Sampling and decoding

How logits become a chosen token: temperature, top-k and top-p, why pure sampling degenerates, and what to do when providers remove the knobs.

UPDATED 18 NOV 2026#llm#sampling#temperature#top-p#decoding

From scores to a token

At each step the model outputs a score (a logit) for every token in its vocabulary. A decoding strategy turns those scores into one chosen token. Run it in a loop and you get text.

Diagram: logits are divided by temperature, passed through softmax, trimmed with top-k or top-p, then one token is sampled

Diagram: my own.

The simplest strategy is greedy: always take the highest-scoring token. It is deterministic, but for open-ended text it tends to be repetitive. Sampling from the full distribution fixes repetition but brings a different problem.

Why pure sampling fails

Holtzman and colleagues studied text degeneration, where output is bland, incoherent or stuck in repetitive loops1. They found that decoding strategy alone can dramatically change text quality, even from exactly the same model1. Their diagnosis for pure sampling is the “unreliable tail”: many individually unlikely tokens that together hold enough probability to be picked often, derailing the text1.

Their remedy, nucleus (top-p) sampling, samples only from the smallest set of tokens whose cumulative probability reaches p, which the authors call the nucleus. That set expands and contracts as the distribution changes1. They report it as the best of the strategies compared for long-form text that is both high quality and diverse1.

Diagram: tokens sorted by probability with the smallest set reaching 90 percent kept and the tail dropped

Diagram: my own. Probabilities come from an example set of logits.

Top-k, by contrast, keeps a fixed number of top tokens regardless of the distribution’s shape. When the model is very sure, k may include junk; when it is unsure, k may cut good options1.

Temperature

Temperature rescales the logits before softmax. Low temperature sharpens the distribution toward the top token; high temperature flattens it. Temperature near zero approaches greedy decoding.

Diagram: the same five logits at low, medium and high temperature, showing probability concentrating or spreading

Diagram: my own, computed from the logits 3.0, 2.2, 1.6, 0.8, 0.2 at temperatures 0.5, 1 and 2.

Holtzman et al. also note that lowering temperature reduces the tail problem but costs diversity and brings repetition1, which is why truncation methods like top-p were proposed alongside it.

Practical settings

My working defaults, from general experience rather than the sources:

  • Extraction, classification, code, tool arguments: low temperature, for consistency.
  • Brainstorming, varied phrasing: moderate to higher temperature.
  • Change one knob at a time. Temperature and top-p both shape the same distribution.
  • Do not rely on determinism. Even at zero temperature, outputs can vary across hardware and batching, so treat equality tests with care and use evals rather than exact matching.

The knobs may not exist

Providers are retiring these controls on newer models. One compatibility reference reports that on Claude Opus 4.7 and later, a non-default temperature, top_p or top_k returns a 400 error, and that Google has deprecated them on newer Gemini models2. This is a secondary source and moves fast, so check the current API docs for your model. When sampling controls are gone, steer behaviour through the prompt, structured outputs, and reasoning effort instead (see Reasoning models).

Sources

  1. [1]The Curious Case of Neural Text Degeneration — Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, Yejin Choi · arXiv, ICLR 2020 · 2019PARAPHRASED
  2. [2]LLM API Parameter Compatibility Reference — Hidekazu Konishi · hidekazu-konishi.com · 2026PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.