threatdeck
Live 13:45 UTC ·

Learn / Red teaming AI systems

Lesson 2 of 69 minReviewed

Scoping an AI engagement and its rules of engagement

Describe the system under test down to its tools, data and identities, write a threat model with impacts that count, and agree authorisation, test data and stop conditions before the first prompt.

Most failed AI engagements fail at scoping. The team tests a model endpoint, finds that it can be talked into rude answers, and misses the tool that can send email on any user's behalf. Scope the system as it is deployed, then decide which outcomes count.

Describe the system under test

  • Model and version, with the provider and any fine-tune. A change of version reopens the scope.
  • System prompt and orchestration: how the context is assembled, conversation history, memory.
  • Retrieval sources: which indexes, mailboxes, drives or web domains the system reads, and who can write to them.
  • Tools and their permissions: every function or MCP server, the identity it runs as, and what it can change.
  • Output paths: where model output is rendered or executed (browser, markdown, shell, another agent).
  • Guardrails and filters in front of or behind the model.

Microsoft's planning guidance separates testing the base model with its safety system, usually through an API, from testing the application, best done through its UI, and recommends testing both with and without mitigations in place. Reports should say which endpoint each result came from [1]. The OWASP GenAI Red Teaming Guide's four areas (model, implementation, infrastructure, runtime behaviour) are a useful completeness check for this list [2].

Threat model: who, what, and what counts

Write down attacker positions explicitly. An outsider who can only get text into a document, email or web page the system reads is a different attacker from a signed-in user, and both differ from an insider who can edit the system prompt. Then list the assets (customer records, credentials, money movement, code repositories) and the impacts that will be reported as findings: disclosure across users, actions taken without the user's intent, persistence in memory, cost abuse. Tag each with an OWASP or ATLAS id, such as LLM06 for excessive agency, so results map to the same vocabulary as the rest of the programme [3].

Rules of engagement

Rules of engagement checklist for an AI system
ItemWhat to agree
AuthorisationWritten approval from the system owner, naming environments, accounts and dates
Third partiesWhich external systems agents may touch. An agent that sends mail or calls a SaaS API reaches real people; use test tenants and owned recipients
Provider and cloud termsRead the model provider's and cloud provider's testing policies. AWS, for example, lists services testable without prior approval and prohibits denial of service and request flooding [4]
Test dataSynthetic records and unique canary strings in place of real secrets or personal data, so a leak is unambiguous and harmless
Stop conditionsHalt and notify on real data exposure, unexpected production changes or a token-spend ceiling
RecordsFor every example: date, a unique id for the input/output pair, the input, and the output or a screenshot [1]
HandlingWhere transcripts and reproduction steps are stored, who can read them, and when they are deleted

NIST AI 600-1 asks organisations to document the instructions given to AI red-teamers (MS-2.8-002) [5]. The same document makes the engagement repeatable: Microsoft recommends giving testers the purpose of the round, the features in scope and how to access them, the kinds of issues to look for, the time to spend, how to record results and whom to contact [1].

With scope, threat model and rules agreed, testing can start. The next lesson covers the hardest part of most scopes: agents that read untrusted content and act on it.

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. A team plans to red-team a support agent by sending prompts to the model provider's API with the system prompt pasted in. What does this miss most?
  2. Why seed test data with unique canary strings instead of using a copy of production records?
  3. Which belongs in the rules of engagement for an agent that can send email?
  4. Microsoft's guidance says each recorded example should include which minimum fields?

References

  1. Planning red teaming for large language models (LLMs) and their applications. Microsoft Learn, 2026-05.
  2. GenAI Red Teaming Guide. OWASP GenAI Security Project, 2025-01.
  3. LLM06:2025 Excessive Agency. OWASP GenAI Security Project, 2024-11.
  4. Penetration Testing policy. Amazon Web Services.
  5. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.