threatdeck
Live 13:45 UTC ·

Learn / Red teaming AI systems

Lesson 1 of 68 minReviewed

What AI red teaming is, and what it is not

Define AI red teaming for a security team, separate security findings from safety benchmarking, and place the work in NIST AI 600-1 and the OWASP GenAI Red Teaming Guide.

The phrase "red teaming" has stretched. Microsoft's planning guidance notes that it began as systematic adversarial testing for security flaws and now describes many kinds of probing, testing and attacking of AI systems [5]. NIST AI 600-1 defines AI red-teaming as a structured exercise that probes an AI system for flaws and vulnerabilities, usually in a controlled environment and with the developers involved [3]. For a security team a narrower working definition helps: try to make one specific AI system, as deployed with its data, tools and users, do something its owner would treat as a security incident, and show how it happened.

Security and safety ask different questions

Two kinds of finding that often share one report
Security red teamingSafety and responsible-AI testing
Typical findingAn injected document makes the agent mail another customer's invoices to an outside addressThe model produces disallowed or biased content when asked
Who is harmedThe system owner, its users and third parties whose data or accounts are touchedThe user who sees the output, or people it describes
What counts as successAn attacker achieves an action or disclosure across a trust boundaryOutput violates a content or quality policy
Who fixes itApplication and platform engineers: permissions, approvals, egress, output handlingModel, policy and content-filter owners

Both matter, and one team may do both. Mixing them in a single list without labels produces reports that nobody owns. Microsoft's AI red team, writing up work on more than 100 generative AI products, lists "AI red teaming is not safety benchmarking" as one of its eight lessons [1]. A benchmark measures a rate on a fixed dataset; a red team looks for failures that matter in one system's context, including ones nobody has written a test for yet.

What 100 products taught one team

Several of the eight lessons [1] translate directly into planning decisions:

  • Start from what the system can do and where it is applied. Capability and deployment context decide which failures are worth an engagement.
  • You don't have to compute gradients to break an AI system. Most real-world attacks are crafted inputs through the normal interface, not optimisation against model weights.
  • LLMs amplify existing security risks and introduce new ones. Outdated dependencies and poor credential handling still apply alongside prompt injection [2].
  • Automation widens coverage; people stay in the loop. Two of the lessons pair these ideas, and Microsoft's advice is to use tools such as PyRIT to scale and keep people in the loop for judgment and context [2].
  • The work of securing AI systems will never be complete. Mitigations reduce risk; the testing, measurement and mitigation cycle repeats [1, 2].
The AI red teaming engagement loop Six steps in a cycle: scope the system under test, threat model attackers and impacts, test manually and with automation, triage findings by achieved impact, fix in deterministic controls, and add each finding to a regression suite. A change to the model, prompt, tools or data restarts the loop at scoping. 1. Scope system, tools, data, rules 2. Threat model attackers, assets, impacts 3. Test manual rounds + automation 4. Triage by impact what did the attacker achieve? 5. Fix in deterministic controls 6. Regression test every finding becomes a case model, prompt, tool or data change
An engagement is a loop, not a one-off test: each change to the model, prompt, tools or data reopens scoping.

Where it sits in the frameworks

NIST AI 600-1 makes red teaming an explicit measurement action: MS-2.7-007 calls for AI red-teaming to assess resilience against abuse that facilitates attacks on other systems, generative AI attacks such as prompt injection, and machine learning attacks such as adversarial examples [3]. The profile also describes red teams of general-public testers, domain experts, a combination, and human teams working with generative AI, and notes that quality depends on the team's expertise [3].

The OWASP GenAI Red Teaming Guide organises the work into four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis [4]. Use those areas as a coverage check: an engagement that only sends prompts to a model endpoint has skipped three of them.

The rest of this path covers scoping, testing agents, choosing tools, writing findings by impact and keeping the tests running after the engagement ends.

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. Which result is a security red teaming finding rather than a safety benchmarking result?
  2. Microsoft's lesson "You don't have to compute gradients to break an AI system" mainly tells a planner to:
  3. According to NIST AI 600-1 action MS-2.7-007, AI red-teaming should assess resilience against which set?
  4. An engagement consisted only of sending prompts to a model's API endpoint. Using the OWASP guide's four areas, what is missing?

References

  1. Lessons From Red Teaming 100 Generative AI Products. Bullwinkel et al., Microsoft (arXiv:2501.07238), 2025-01.
  2. 3 takeaways from red teaming 100 generative AI products. Microsoft Security Blog, 2025-01-13.
  3. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
  4. GenAI Red Teaming Guide. OWASP GenAI Security Project, 2025-01.
  5. Planning red teaming for large language models (LLMs) and their applications. Microsoft Learn, 2026-05.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.