Capabilities and failure modes
What models are good at, and the failures to design around: hallucination, drifting over long inputs, prompt sensitivity, reasoning slips and unsafe actions.
What they are good at
Trained on next-token prediction at scale, language models handle fluent writing, summarising, translating, classifying, extracting structure from text, and writing and explaining code. They can perform new tasks from instructions or a few examples in the prompt, without retraining4. Combined with tools, they can act on the world. The same training that makes them flexible also produces predictable weaknesses.
Diagram: my own map from failure to habit.
1. Hallucination
Models produce plausible but false statements. The OpenAI and Georgia Tech paper argues this happens because training and evaluation procedures reward guessing over acknowledging uncertainty, like students guessing on exam questions1. They analyse hallucination as an ordinary statistical error: if invalid statements cannot be told apart from facts, errors will arise through natural statistical pressure in pretraining1. They also argue it persists because most benchmarks are graded in a way that makes guessing pay1.
Diagram: my own illustration of the incentive argument in 1.
Their suggested fix is socio-technical: change the scoring of existing influential benchmarks so uncertainty is not penalised, rather than add more hallucination tests1. For builders: ground answers in retrieved or tool-provided sources (RAG overview), ask for citations, allow “I don’t know” in the prompt, and verify. For a wider survey, see Ji et al.3
2. Weak use of long context
Bigger windows do not mean uniform attention. Liu and colleagues tested multi-document question answering and key-value retrieval and found performance can degrade significantly when the position of the relevant information changes. It is best at the start or end of the input and worst in the middle, and it decreases as context grows longer, even for models built for long contexts2.
Diagram: my own and illustrative, showing the shape reported in 2. Models have changed since 2023, so measure your own.
Implications: keep prompts lean, put key instructions and evidence where they will be used, and retrieve rather than dump. See Long-context trade-offs.
3. Sensitivity to wording and format
Small changes in phrasing, ordering or examples can shift results. Treat prompts as code: version them and test variants against an eval set. Prompt design covers structure.
4. Reasoning slips
Even strong models make arithmetic and logic mistakes, and can talk themselves into wrong answers. Letting the model think helps (Reasoning models), but verifying with code, tests or a checker helps more. GPT-3’s authors already noted tasks where few-shot learning struggled4.
5. Wrong or unsafe actions
When models act through tools, errors become side effects. Limit permissions, sandbox execution, and require approval for irreversible steps (Human in the loop, Permissions).
A habit that covers all five
Assume the model will sometimes be wrong, and build the system so that wrongness is detected (evals, tracing), contained (limits, approvals) and recoverable (error handling).
Sources
- [1]Why Language Models Hallucinate — Adam Tauman Kalai, Ofir Nachum, Santosh S. Vempala, Edwin Zhang · OpenAI and Georgia Tech · 2025PARAPHRASED
- [2]Lost in the Middle: How Language Models Use Long Contexts — Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, Percy Liang · arXiv, TACL · 2023PARAPHRASED
- [3]Survey of Hallucination in Natural Language Generation — Ziwei Ji and co-authors · arXiv · 2022FURTHER READING
- [4]Language Models are Few-Shot Learners — Tom B. Brown and co-authors · arXiv, NeurIPS 2020 · 2020PARAPHRASED