Path 5 of 6Practitioner6 lessons · 50 minReviewed 2026-11-10
Red teaming AI systems
Plan and run security red team engagements against LLM applications and agents: scoping and rules of engagement, agent and indirect-injection testing, open-source tooling, impact-based findings and continuous regression testing.
For: Penetration testers, AppSec engineers and internal red teams adding AI systems to scope
- 01 What AI red teaming is, and what it is not
Define AI red teaming for a security team, separate security findings from safety benchmarking, and place the work in NIST AI 600-1 and the OWASP GenAI Red Teaming Guide.
8 min · 4 questions
- 02 Scoping an AI engagement and its rules of engagement
Describe the system under test down to its tools, data and identities, write a threat model with impacts that count, and agree authorisation, test data and stop conditions before the first prompt.
9 min · 4 questions
- 03 Testing agents and indirect prompt injection
Design agent tests where the attacker controls a tool output and the goal is an action, measure utility and attack success separately, and avoid the three ways agent results mislead: single attempts, aggregates and static attacks.
9 min · 4 questions
- 04 Choosing red team tooling: garak, PyRIT, promptfoo, Inspect
Match open-source tools to the job (scanning an endpoint, orchestrating multi-turn campaigns, testing an application in CI, building custom evaluations) and know the limits of automated results.
8 min · 4 questions
- 05 Impact-based findings and reporting
Rate AI findings by what the attacker achieved, make nondeterministic results reproducible, map them to OWASP and ATLAS ids, and hand over a fix plus a regression test.
8 min · 4 questions
- 06 Running it continuously: regression suites and retest triggers
Turn findings into a regression suite, decide which changes trigger a retest, gate CI on results without flaky failures, and keep periodic human rounds for what automation misses.
8 min · 4 questions
Lessons cite primary sources and were last checked on the date shown in each lesson. Corrections are welcome by email (footer). Other paths: Fundamentals · LLM applications · Agents & MCP · AI coding assistants · Detect & respond