NOTES / LLM FUNDAMENTALS

Tokens and tokenisation

How text becomes the integers a model reads: subword tokens, byte pair encoding, and why token counts differ between models and from word counts.

UPDATED 18 NOV 2026#llm#tokens#bpe#tokenizer

Why tokens exist

A language model works on numbers, so text has to be converted first. The unit of conversion is the token: a chunk of text that might be a whole word, part of a word, or a single character or symbol. Whole-word vocabularies fail on rare and unseen words. Sennrich and colleagues addressed this for machine translation by encoding rare and unknown words as sequences of subword units, which let a fixed-size vocabulary represent an open-ended set of words1.

Diagram: text split into tokens, mapped to integer IDs, then to vectors

Diagram: my own. The splits and IDs are illustrative only; each model has its own vocabulary.

The model never sees letters or words directly. It sees a sequence of token IDs, each looked up in a table of learned vectors. That is why models can be oddly bad at things like counting letters in a word: the letters were never the unit.

Byte pair encoding

The method most subword tokenisers descend from is byte pair encoding (BPE). It began as a compression technique that repeatedly replaces the most frequent pair of bytes with a new symbol. Sennrich et al. adapted it to words by starting from characters and iteratively merging the most frequent adjacent pair into a new symbol, so that frequent character sequences, up to whole common words, become single symbols1. The number of merges decides the final vocabulary size1.

Diagram: the word lower starts as five characters, then merges e and r, then l and o, then lo and w

Diagram: my own, a toy walk-through of the merge loop described in 1. Real merges are learned from a large corpus.

The result is a vocabulary where common words are one token and rare words split into several pieces. In their translation experiments, subword models beat a back-off dictionary baseline by 1.1 and 1.3 BLEU on two language pairs1.

Counting tokens in practice

Token counts drive cost, speed and what fits in the context window. Three habits help.

  1. Count with the model’s own tokenizer. Anthropic’s token counting endpoint counts using the tokenizer of the model you pass, so you can compare models by counting the same request against each2.
  2. Expect differences between generations. Anthropic’s documentation says that newer models share a tokenizer that produces roughly 30 percent more tokens for the same text than models before Claude Opus 4.7, with the exact increase depending on content2.
  3. Count the whole request. The endpoint accepts system prompts, tools, images and PDFs, not just the user message2. Treat the result as an estimate; actual usage can differ slightly2.

Diagram: the same text yields about thirty percent more tokens under a newer tokenizer

Diagram: my own, based on the difference described in 2. Bar heights are proportional to the stated ratio, not measured on any text.

Consequences for builders

  • Cost and latency scale with tokens, not words. Languages and content types such as code or numbers can tokenise less efficiently than plain English.
  • Context budgets move when you change models. A prompt that fit may not after a tokenizer change, so re-count when upgrading.
  • Formatting is not free. Verbose JSON keys, long tool definitions and repeated boilerplate all spend tokens. See Latency and cost and Prompt caching.
  • Do not trust rules of thumb. “Four characters per token” is a rough English average at best. Use the counter.

Next: how the model uses these tokens, in Transformers and attention.

Sources

  1. [1]Neural Machine Translation of Rare Words with Subword Units — Rico Sennrich, Barry Haddow, Alexandra Birch · arXiv, ACL 2016 · 2015PARAPHRASED
  2. [2]Token counting — Anthropic, Claude Platform docsPARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.