NOTES / FINE-TUNING AND ADAPTATION
Preparing training data
Data decides the result. Collecting examples, deduplication, filtering, splitting without leakage, synthetic data and its limits, and a checklist before you train.
Quality beats quantity
LIMA is the standard reminder. A model tuned on 1,000 curated examples matched or beat one tuned on 52,000 machine-generated examples in the authors’ human study1. The details differ for narrow tasks, but the habit holds: spend your time on the data.
AWS’s fine-tuning lifecycle puts use-case definition first and data preparation before training7. I would add an eval set before both.
Where examples come from
- Production logs, after consent and redaction: real inputs, with outputs corrected by people. Often the best source.
- Expert-written demonstrations: costly, high quality, good for small sets.
- Existing records, such as tickets or reviewed documents, converted into input-output pairs.
- Synthetic data generated by a model, then checked (Distillation).
If the data contains personal or regulated information, handle it as such before it reaches a training job (Privacy and regulation).
The pipeline
Diagram: my own.
- Deduplicate. Lee et al. found that standard text datasets contain many near-duplicates and that training on deduplicated data made models emit memorised text about ten times less often, and reduced train-test overlap2. The same logic applies at small scale: repeated examples over-weight one case and inflate scores.
- Filter. Remove broken, truncated, off-topic, unsafe or personal data. Keep a log of what was removed and why.
- Review a sample by hand. Read 50 to 100 random examples. You will find problems that no script flags.
- Normalise the format. Consistent roles, templates and stop tokens. Check that your trainer applies the right chat template and masks the right tokens4 (Supervised fine-tuning).
- Split by source. See below.
- Freeze the test set and never tune on it.
Splitting without leakage
Diagram: my own.
Random row splits put near-identical examples on both sides. Split by the unit that should not cross: conversation, customer, document, patient or date. Then deduplicate across splits. Lee et al. report that train-test overlap was common in non-deduplicated datasets2. In time-ordered data, test on later dates to mimic deployment.
Synthetic data, carefully
Self-Instruct generated about 52,000 instructions with 82,000 instances from a small seed set, then filtered them5. It shows that synthetic data can work, and that generation is paired with filtering. Risks:
- Inherited errors: the generator’s mistakes become the student’s.
- Narrow diversity: generated prompts cluster. Vary topics, difficulty and phrasing.
- Recursive drift: Shumailov et al. found that training on model-generated data causes the tails of the distribution to vanish over generations3. Keep real examples in the mix.
Facts versus behaviour
If your examples mostly encode facts, reconsider. Ovadia et al. found that fine-tuning learned new facts poorly and that retrieval beat it, though exposing the model to many variations of the same fact helped6. Use retrieval for facts and keep tuning data about behaviour.
Checklist before training
- [ ] Eval set built from real cases, separate from training data
- [ ] Duplicates and near-duplicates removed, including across splits
- [ ] Sample read by a human; issues logged
- [ ] Format and template verified on a decoded batch
- [ ] Personal data handled; licences and provider terms checked
- [ ] Baseline score from the best prompted model recorded
- [ ] Stopping rule defined: which metric, on which split
Then train, and judge the result on the frozen test set (Building eval sets).
Sources
- [1]LIMA: Less Is More for Alignment — Zhou et al. · 2023PARAPHRASED
- [2]Deduplicating Training Data Makes Language Models Better — Lee et al. · 2022PARAPHRASED
- [3]The Curse of Recursion: Training on Generated Data Makes Models Forget — Shumailov et al. · 2023PARAPHRASED
- [4]SFT Trainer — Hugging Face TRL documentationPARAPHRASED
- [5]Self-Instruct: Aligning Language Models with Self-Generated Instructions — Wang et al. · 2023PARAPHRASED
- [6]Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs — Ovadia et al. · 2024PARAPHRASED
- [7]Best practices and lessons for fine-tuning Anthropic's Claude 3 Haiku on Amazon Bedrock — AWS Machine Learning Blog · 2024FINE-TUNING LIFECYCLE