NOTES / FINE-TUNING AND ADAPTATION
Distillation
Training a small student to imitate a large teacher: classic soft-target distillation, text-based distillation from API models, and the legal and quality traps.
The idea
A large model can be expensive to run. Distillation trains a smaller student to reproduce the behaviour of a larger teacher, aiming to keep most of the quality at lower cost and latency (Latency and cost).
Classic distillation
Hinton and colleagues proposed using the teacher’s class probabilities as soft targets for the student. When those soft targets have high entropy they carry much more information per training case than hard labels, which lets the student learn how the teacher generalises1. A temperature parameter softens the probabilities during training1.
DistilBERT applied this during pre-training and reported a model 40 percent smaller and 60 percent faster that kept about 97 percent of BERT’s language-understanding performance on their benchmark2. Those figures are the authors’ and apply to that model and task set.
This approach needs access to the teacher’s output probabilities, so it fits open-weight teachers.
Text-based distillation
With an API model you usually get text, not probabilities. The common recipe is to have the teacher generate training examples and then run supervised fine-tuning on the student. Self-Instruct showed how a model can bootstrap instructions and responses from its own generations, starting from a small human-written seed set4. Stanford’s Alpaca applied this: it started with 175 human-written seed pairs, used text-davinci-003 to generate 52,000 instruction-response examples at a reported cost of under $500, and tuned LLaMA 7B on them3.
Diagram: my own.
A practical pipeline
Diagram: my own.
- Use real prompts from your task, not generic ones, so the student sees the distribution it will face.
- Generate answers from the teacher, possibly several per prompt.
- Filter and verify. Drop wrong or low-quality outputs with checkers, rules or a judge model (LLM-as-judge). The student inherits the teacher’s mistakes.
- Train with SFT, often with adapters (LoRA and QLoRA).
- Compare student and teacher on a held-out eval set, per category, and set a quality floor before switching traffic.
Traps
- Provider terms. Alpaca’s authors noted that the data came from a model whose terms prohibit developing competing models, and that LLaMA’s licence was non-commercial, which is why they restricted use to research3. A commentary reports that major providers include similar conditions6. Read the current terms of any provider before using its outputs as training data. I am not a lawyer; ask one if this matters to you.
- Recursive degradation. Shumailov et al. found that models trained on model-generated data lose the tails of the original distribution, and that access to real data is needed to avoid this5. Mix in real, human-verified examples and avoid chaining students on students.
- Narrow gains. A student tuned on one task may do worse elsewhere.
- Overstated parity. Averages hide the hard cases. Look at slices.
When it pays
Distillation pays when volume is high, the task is narrow, and a quality floor can be measured. It is a late step on the ladder in When to fine-tune, not a first one.
Sources
- [1]Distilling the Knowledge in a Neural Network — Hinton, Vinyals, Dean · 2015PARAPHRASED
- [2]DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter — Sanh, Debut, Chaumond, Wolf · 2019PARAPHRASED
- [3]Alpaca: A Strong, Replicable Instruction-Following Model — Stanford CRFM · 2023PARAPHRASED
- [4]Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · 2023PARAPHRASED
- [5]The Curse of Recursion: Training on Generated Data Makes Models Forget — Shumailov et al. · 2023PARAPHRASED
- [6]When Does Distilling GPT Become Theft? — Legal Layer (Substack)COMMENTARY ON PROVIDER TERMS; NOT LEGAL ADVICE