Red teaming
Attack your own agent before others do: scope, manual and automated methods, adaptive attackers, and turning every finding into a regression test.
What it is
Red teaming means deliberately trying to make your system fail in the ways an adversary or an unlucky user would. For agents the goals include getting the agent to leak data, take a forbidden action, ignore its instructions, or loop and burn resources.
NIST’s generative AI profile suggests adversarial role-playing, red teaming and chaos testing to find failure modes nobody anticipated, and asks organisations to decide where red teaming would help most for their context2. It is voluntary guidance rather than a certification2.
Scope it from your threat model
Start from the threat model: which assets, which entry points, which tools. Write attack goals as testable statements, for example “an email can cause the agent to send my calendar to a third party”. The OWASP agentic list is a useful prompt for coverage, since it catalogues risks for systems that plan, use tools and act5.
Methods
Manual attacks. People with an adversarial mindset try to break the system. They find creative failures that scripts miss. Include people who know the domain.
Automated generation. Perez and colleagues showed that one language model can generate test cases for another, with a classifier detecting offensive replies, and used the approach to uncover tens of thousands of offensive replies in a large chatbot1. The same idea now extends to injection and tool misuse: have a model write attacks, run them, score the outcomes.
Adaptive attackers. A fixed list of attacks goes stale. Anthropic evaluates browser-agent defences against an internal adaptive attacker that tries and combines many attack techniques, and reports success rates under that attacker rather than against a static list4. When you measure a defence, let the attacker see it and respond.
Sustained external testing. Anthropic’s classifier work reports more than 3,000 estimated hours of red teaming, during which no tester found a universal jailbreak against an early guarded system at similar detail to an unguarded one3. Bounty programmes are a way to buy this kind of effort.
The loop
Diagram: my own.
- Set goals from the threat model.
- Attack with a mix of manual and automated methods.
- Record failures with the full input and a trace, so they can be reproduced (Tracing and monitoring).
- Fix at the right layer: prompt, tool design, permission, sandbox or guardrail. Prefer changes that limit what a hijacked agent can do over changes that only make a specific attack fail.
- Add the case to your regression set (Building eval sets) so it never silently returns.
- Repeat on every change to the model, prompt, tools or data.
Measuring
- Attack success rate per attack category, under an adaptive attacker.
- Severity, not only count: one data leak outweighs many harmless refusals.
- Over-refusal rate on legitimate tasks, so defences are not tuned to uselessness.
- Time to detect in monitoring, since some attacks will succeed.
Report results with their conditions. A number without the attacker model, the number of attempts and the defences in place is not comparable.
Cautions
- Absence of findings is not safety. It means your testers did not find a problem with their methods and time.
- Vendor figures describe the vendor’s setup. Use them as signals, then test your own system.
- Keep records secure. Your attack library and findings are sensitive.
- Stay in scope and in law. Test only systems and data you are authorised to test.
Red teaming closes the loop with Why evals: the attacks you find become part of the tests you run forever.
Sources
- [1]Red Teaming Language Models with Language Models — Perez et al., DeepMind · 2022PARAPHRASED
- [2]NIST AI 600-1 Generative AI Profile guide — Anaconda (summary of NIST AI 600-1)OVERVIEW OF THE PROFILE, AS SUMMARISED BY A THIRD PARTY
- [3]Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — Sharma et al., Anthropic · 2025PARAPHRASED
- [4]Mitigating the risk of prompt injections in browser use — Anthropic · 2025PARAPHRASED
- [5]OWASP Top 10 for Agentic Applications for 2026 — OWASP GenAI Security Project · 2025PARAPHRASED