NOTES / LLM FUNDAMENTALS

How models are trained

From next-token prediction to a useful assistant: pretraining, scaling laws, instruction tuning with human feedback, and reinforcement learning for reasoning.

UPDATED 18 NOV 2026#llm#pretraining#rlhf#scaling-laws

Stage one: predict the next token

A base model is trained to predict the next token of text. For each position in a huge text corpus the model outputs a probability distribution over the vocabulary, and training nudges its weights to make the real next token more likely.

Diagram: context so far goes into the model, which outputs next-token probabilities; the loss measures how unlikely the actual token was and weights are updated

Diagram: my own.

This simple objective yields surprising breadth. The GPT-3 paper trained an autoregressive model with 175 billion parameters and showed it could perform tasks from a few examples placed in the prompt, with no gradient updates or fine-tuning. The task and demonstrations were specified purely through text1. The authors called this in-context learning1, which is why prompt examples work (see Few-shot examples).

Scale: parameters and tokens

Given a fixed compute budget, how big should the model be, and how much data should it see? DeepMind trained over 400 models, from 70 million to over 16 billion parameters, on 5 to 500 billion tokens. They concluded that model size and training tokens should scale together: each doubling of model size should come with a doubling of training tokens2. They argued that large models of the time were significantly undertrained2.

They tested it with Chinchilla, a 70B model trained on 1.4 trillion tokens with the same compute as the 280B Gopher, and it outperformed Gopher, GPT-3 and others across many evaluations2.

Diagram: GPT-3 with 175 billion parameters on 300 billion tokens versus Chinchilla with 70 billion parameters on 1.4 trillion tokens

Diagram: my own, using the model sizes and token counts reported in 2.

The lesson for practitioners is that a smaller, well-trained model can beat a larger, under-trained one, and a smaller model is also cheaper to serve. Later practice often trains even longer than this compute-optimal point to get cheaper inference, but that is my general knowledge rather than a claim from these sources.

Stage two: make it follow instructions

A base model completes text; it does not reliably do what you ask. The InstructGPT work reshaped GPT-3 into an assistant using three steps: supervised fine-tuning on human-written demonstrations, training a reward model from human rankings of outputs, and reinforcement learning with PPO against that reward model3. Human raters preferred outputs from the 1.3B InstructGPT model to those from the 175B GPT-3, despite it having 100 times fewer parameters3.

Diagram: four stages, pretraining, supervised fine-tuning, preference training and reinforcement learning

Diagram: my own. The last three stages follow the recipe in 3.

Detail on these stages from the practitioner’s side is in Supervised fine-tuning and Preference tuning.

Stage three: reinforcement learning for reasoning

More recent work applies reinforcement learning to improve reasoning directly. DeepSeek reported that reasoning abilities can be incentivised through pure reinforcement learning, without human-labelled reasoning trajectories. Their R1-Zero model was trained with large-scale RL and no supervised fine-tuning step, but had issues such as poor readability and language mixing. R1 added a small amount of cold-start data and a multi-stage pipeline to address these4. They also found that distilling from the larger model into a smaller one beat applying RL directly to the smaller model4. The behaviour this produces is covered in Reasoning models, and distillation in Distillation.

What this means when you build

  • Behaviour comes from several layers. Base capability from pretraining, manners from instruction tuning, extra reasoning from RL. Prompts only steer the result.
  • Training data sets limits. A model cannot reliably know what was rare or absent in training, which links to Capabilities and failure modes.
  • Before fine-tuning, ask whether you need to. See When to fine-tune.

Sources

  1. [1]Language Models are Few-Shot Learners — Tom B. Brown and co-authors · arXiv, NeurIPS 2020 · 2020PARAPHRASED
  2. [2]Training Compute-Optimal Large Language Models — Jordan Hoffmann and co-authors · arXiv, DeepMind · 2022PARAPHRASED
  3. [3]Training language models to follow instructions with human feedback — Long Ouyang and co-authors · arXiv, NeurIPS 2022 · 2022PARAPHRASED
  4. [4]DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning — DeepSeek-AI · arXiv · 2025PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.