NOTES / PROMPTING AND CONTEXT ENGINEERING
Few-shot examples
Showing the model what you want: how in-context examples work, what they actually teach, and how to choose three to five good ones.
What few-shot means
Instead of describing a task, you place a few input and output pairs in the prompt and let the model continue the pattern. The GPT-3 paper showed that a large model could do tasks this way, with the task and demonstrations specified purely as text and no gradient updates1. It is called in-context learning (see How models are trained).
What examples actually teach
It is natural to assume the model learns the right answers from your examples. Min and colleagues tested this across 12 models including GPT-3 and found that ground-truth labels were not required: randomly replacing the labels in the demonstrations barely hurt performance2. What mattered instead was that demonstrations show the label space, the distribution of the input text, and the overall format of the sequence2.
Diagram: my own, summarising the finding in 2.
Practical reading: examples are strongest as a way to set format, scope and style. Do not expect a handful of examples to teach new facts. Those findings come from classification and multiple-choice tasks in older models, so test on your own task before leaning on them.
Choosing good examples
Anthropic’s guide recommends three to five examples, and says they should be relevant to the real use case, diverse enough to cover edge cases without inviting accidental patterns, and clear, wrapped in example tags nested in an examples tag3. It also suggests asking the model to critique your example set for relevance and diversity, or to generate more from your seed set3.
Diagram: my own, following the guidance in 3.
A warning from Anthropic’s context engineering post: stuffing a prompt with a long list of edge cases to cover every rule is a common mistake. Prefer a small set of diverse, canonical examples that portray expected behaviour4.
Pitfalls
- Accidental patterns. If every example ends the same way, the model may copy that, not the rule. Vary length, order and difficulty.
- Leaking test data. Never reuse eval items as examples, or scores will flatter you.
- Cost. Examples are tokens on every call. They are good candidates for prompt caching.
- Dynamic selection. For large pools, retrieve the examples most similar to the current input rather than hard-coding them. That is a retrieval problem; see RAG overview.
- When examples are not enough, consider supervised fine-tuning.
Next: Structured outputs.
Sources
- [1]Language Models are Few-Shot Learners — Tom B. Brown and co-authors · arXiv, NeurIPS 2020 · 2020PARAPHRASED
- [2]Rethinking the Role of Demonstrations: What Makes In-Context Learning Work? — Sewon Min, Xinxi Lyu, Ari Holtzman, Mikel Artetxe, Mike Lewis, Hannaneh Hajishirzi, Luke Zettlemoyer · arXiv, EMNLP 2022 · 2022PARAPHRASED
- [3]Prompting best practices — Anthropic, Claude Platform docsPARAPHRASED
- [4]Effective context engineering for AI agents — Anthropic Engineering · 2025PARAPHRASED