NOTES / PART 3 · RUNNING IN PRODUCTION

11Fine-tuning and adaptation

Changing the model itself, and when that beats prompting or RAG.

THIS SECTION COVERS

When to fine-tune, LoRA and QLoRA, supervised fine-tuning, preference tuning (RLHF, DPO), distillation, data preparation.

Related: Inference and serving · Evals and observability

TOPICS · 6/6 WRITTEN

  1. 11.1When to fine-tuneFine-tuning changes how a model behaves, not what it can look up. A decision ladder, the evidence behind it, and what to measure before and after.
  2. 11.2Supervised fine-tuningTraining on example conversations with a next-token loss: data formats, loss masking on assistant tokens, how much data you need, and the common mistakes.
  3. 11.3LoRA and QLoRAParameter-efficient fine-tuning: train small low-rank adapters on a frozen model, and in QLoRA keep the frozen model in 4 bits. What it saves, and what it costs.
  4. 11.4Preference tuning: RLHF and DPOHow models are tuned from comparisons rather than single right answers: the RLHF pipeline, DPO's shortcut, and reinforcement learning with verifiable rewards.
  5. 11.5DistillationTraining a small student to imitate a large teacher: classic soft-target distillation, text-based distillation from API models, and the legal and quality traps.
  6. 11.6Preparing training dataData decides the result. Collecting examples, deduplication, filtering, splitting without leakage, synthetic data and its limits, and a checklist before you train.