NOTES / SAFETY AND GUARDRAILS

Red teaming

Attack your own agent before others do: scope, manual and automated methods, adaptive attackers, and turning every finding into a regression test.

UPDATED 18 NOV 2026#safety#red-teaming#adversarial-testing#evals

What it is

Red teaming means deliberately trying to make your system fail in the ways an adversary or an unlucky user would. For agents the goals include getting the agent to leak data, take a forbidden action, ignore its instructions, or loop and burn resources.

NIST’s generative AI profile suggests adversarial role-playing, red teaming and chaos testing to find failure modes nobody anticipated, and asks organisations to decide where red teaming would help most for their context2. It is voluntary guidance rather than a certification2.

Scope it from your threat model

Start from the threat model: which assets, which entry points, which tools. Write attack goals as testable statements, for example “an email can cause the agent to send my calendar to a third party”. The OWASP agentic list is a useful prompt for coverage, since it catalogues risks for systems that plan, use tools and act5.

Methods

Manual attacks. People with an adversarial mindset try to break the system. They find creative failures that scripts miss. Include people who know the domain.

Automated generation. Perez and colleagues showed that one language model can generate test cases for another, with a classifier detecting offensive replies, and used the approach to uncover tens of thousands of offensive replies in a large chatbot1. The same idea now extends to injection and tool misuse: have a model write attacks, run them, score the outcomes.

Adaptive attackers. A fixed list of attacks goes stale. Anthropic evaluates browser-agent defences against an internal adaptive attacker that tries and combines many attack techniques, and reports success rates under that attacker rather than against a static list4. When you measure a defence, let the attacker see it and respond.

Sustained external testing. Anthropic’s classifier work reports more than 3,000 estimated hours of red teaming, during which no tester found a universal jailbreak against an early guarded system at similar detail to an unguarded one3. Bounty programmes are a way to buy this kind of effort.

The loop

Diagram: set goals, attack, record failures, fix, add to evals, then repeat

Diagram: my own.

  1. Set goals from the threat model.
  2. Attack with a mix of manual and automated methods.
  3. Record failures with the full input and a trace, so they can be reproduced (Tracing and monitoring).
  4. Fix at the right layer: prompt, tool design, permission, sandbox or guardrail. Prefer changes that limit what a hijacked agent can do over changes that only make a specific attack fail.
  5. Add the case to your regression set (Building eval sets) so it never silently returns.
  6. Repeat on every change to the model, prompt, tools or data.

Measuring

  • Attack success rate per attack category, under an adaptive attacker.
  • Severity, not only count: one data leak outweighs many harmless refusals.
  • Over-refusal rate on legitimate tasks, so defences are not tuned to uselessness.
  • Time to detect in monitoring, since some attacks will succeed.

Report results with their conditions. A number without the attacker model, the number of attempts and the defences in place is not comparable.

Cautions

  • Absence of findings is not safety. It means your testers did not find a problem with their methods and time.
  • Vendor figures describe the vendor’s setup. Use them as signals, then test your own system.
  • Keep records secure. Your attack library and findings are sensitive.
  • Stay in scope and in law. Test only systems and data you are authorised to test.

Red teaming closes the loop with Why evals: the attacks you find become part of the tests you run forever.

Sources

  1. [1]Red Teaming Language Models with Language Models — Perez et al., DeepMind · 2022PARAPHRASED
  2. [2]NIST AI 600-1 Generative AI Profile guide — Anaconda (summary of NIST AI 600-1)OVERVIEW OF THE PROFILE, AS SUMMARISED BY A THIRD PARTY
  3. [3]Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming — Sharma et al., Anthropic · 2025PARAPHRASED
  4. [4]Mitigating the risk of prompt injections in browser use — Anthropic · 2025PARAPHRASED
  5. [5]OWASP Top 10 for Agentic Applications for 2026 — OWASP GenAI Security Project · 2025PARAPHRASED

© Ashish Kumar. All rights reserved. Quoted and cited material belongs to its owners. How I attribute sources.

Product, company and framework names are trademarks of their respective owners. Mentioning them does not imply affiliation or endorsement.