Learn / Red teaming AI systems
Lesson 1 of 68 minReviewed
What AI red teaming is, and what it is not
Define AI red teaming for a security team, separate security findings from safety benchmarking, and place the work in NIST AI 600-1 and the OWASP GenAI Red Teaming Guide.
The phrase "red teaming" has stretched. Microsoft's planning guidance notes that it began as systematic adversarial testing for security flaws and now describes many kinds of probing, testing and attacking of AI systems [5]. NIST AI 600-1 defines AI red-teaming as a structured exercise that probes an AI system for flaws and vulnerabilities, usually in a controlled environment and with the developers involved [3]. For a security team a narrower working definition helps: try to make one specific AI system, as deployed with its data, tools and users, do something its owner would treat as a security incident, and show how it happened.
Security and safety ask different questions
| Security red teaming | Safety and responsible-AI testing | |
|---|---|---|
| Typical finding | An injected document makes the agent mail another customer's invoices to an outside address | The model produces disallowed or biased content when asked |
| Who is harmed | The system owner, its users and third parties whose data or accounts are touched | The user who sees the output, or people it describes |
| What counts as success | An attacker achieves an action or disclosure across a trust boundary | Output violates a content or quality policy |
| Who fixes it | Application and platform engineers: permissions, approvals, egress, output handling | Model, policy and content-filter owners |
Both matter, and one team may do both. Mixing them in a single list without labels produces reports that nobody owns. Microsoft's AI red team, writing up work on more than 100 generative AI products, lists "AI red teaming is not safety benchmarking" as one of its eight lessons [1]. A benchmark measures a rate on a fixed dataset; a red team looks for failures that matter in one system's context, including ones nobody has written a test for yet.
What 100 products taught one team
Several of the eight lessons [1] translate directly into planning decisions:
- Start from what the system can do and where it is applied. Capability and deployment context decide which failures are worth an engagement.
- You don't have to compute gradients to break an AI system. Most real-world attacks are crafted inputs through the normal interface, not optimisation against model weights.
- LLMs amplify existing security risks and introduce new ones. Outdated dependencies and poor credential handling still apply alongside prompt injection [2].
- Automation widens coverage; people stay in the loop. Two of the lessons pair these ideas, and Microsoft's advice is to use tools such as PyRIT to scale and keep people in the loop for judgment and context [2].
- The work of securing AI systems will never be complete. Mitigations reduce risk; the testing, measurement and mitigation cycle repeats [1, 2].
Where it sits in the frameworks
NIST AI 600-1 makes red teaming an explicit measurement action: MS-2.7-007 calls for AI red-teaming to assess resilience against abuse that facilitates attacks on other systems, generative AI attacks such as prompt injection, and machine learning attacks such as adversarial examples [3]. The profile also describes red teams of general-public testers, domain experts, a combination, and human teams working with generative AI, and notes that quality depends on the team's expertise [3].
The OWASP GenAI Red Teaming Guide organises the work into four areas: model evaluation, implementation testing, infrastructure assessment and runtime behaviour analysis [4]. Use those areas as a coverage check: an engagement that only sends prompts to a model endpoint has skipped three of them.
The rest of this path covers scoping, testing agents, choosing tools, writing findings by impact and keeping the tests running after the engagement ends.
Try it on ThreatDeck
- Playbook: red-team an LLM application Use the checklist to plan the four OWASP areas for one system you own.
- Labs & practice Pick a deliberately vulnerable app or CTF to practise in an authorised environment.
- Prompt Injection (LLM01) See the technique most security findings start from, with recent real-world stories.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
The calendar case crosses a trust boundary: an outside party causes disclosure through the system's tools. A refusal rate on a fixed dataset is a benchmark measurement, and a wrong answer is a quality or misinformation issue owned by other teams.
-
The lesson is that simple techniques through ordinary interfaces work in practice. Gradient-based attacks need model access that real attackers rarely have, so they should not dominate scope.
-
MS-2.7-007 lists all three categories under information security, which is why a security red team covers misuse of the system against others as well as attacks on the system itself.
-
Prompting the endpoint is model evaluation. The application around it, its infrastructure and its behaviour at runtime with real tools and data are separate areas where most security findings sit.
References
- Lessons From Red Teaming 100 Generative AI Products. Bullwinkel et al., Microsoft (arXiv:2501.07238), 2025-01.
- 3 takeaways from red teaming 100 generative AI products. Microsoft Security Blog, 2025-01-13.
- NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
- GenAI Red Teaming Guide. OWASP GenAI Security Project, 2025-01.
- Planning red teaming for large language models (LLMs) and their applications. Microsoft Learn, 2026-05.