Learn / Red teaming AI systems
Lesson 2 of 69 minReviewed
Scoping an AI engagement and its rules of engagement
Describe the system under test down to its tools, data and identities, write a threat model with impacts that count, and agree authorisation, test data and stop conditions before the first prompt.
Most failed AI engagements fail at scoping. The team tests a model endpoint, finds that it can be talked into rude answers, and misses the tool that can send email on any user's behalf. Scope the system as it is deployed, then decide which outcomes count.
Describe the system under test
- Model and version, with the provider and any fine-tune. A change of version reopens the scope.
- System prompt and orchestration: how the context is assembled, conversation history, memory.
- Retrieval sources: which indexes, mailboxes, drives or web domains the system reads, and who can write to them.
- Tools and their permissions: every function or MCP server, the identity it runs as, and what it can change.
- Output paths: where model output is rendered or executed (browser, markdown, shell, another agent).
- Guardrails and filters in front of or behind the model.
Microsoft's planning guidance separates testing the base model with its safety system, usually through an API, from testing the application, best done through its UI, and recommends testing both with and without mitigations in place. Reports should say which endpoint each result came from [1]. The OWASP GenAI Red Teaming Guide's four areas (model, implementation, infrastructure, runtime behaviour) are a useful completeness check for this list [2].
Threat model: who, what, and what counts
Write down attacker positions explicitly. An outsider who can only get text into a document, email or web page the system reads is a different attacker from a signed-in user, and both differ from an insider who can edit the system prompt. Then list the assets (customer records, credentials, money movement, code repositories) and the impacts that will be reported as findings: disclosure across users, actions taken without the user's intent, persistence in memory, cost abuse. Tag each with an OWASP or ATLAS id, such as LLM06 for excessive agency, so results map to the same vocabulary as the rest of the programme [3].
Rules of engagement
| Item | What to agree |
|---|---|
| Authorisation | Written approval from the system owner, naming environments, accounts and dates |
| Third parties | Which external systems agents may touch. An agent that sends mail or calls a SaaS API reaches real people; use test tenants and owned recipients |
| Provider and cloud terms | Read the model provider's and cloud provider's testing policies. AWS, for example, lists services testable without prior approval and prohibits denial of service and request flooding [4] |
| Test data | Synthetic records and unique canary strings in place of real secrets or personal data, so a leak is unambiguous and harmless |
| Stop conditions | Halt and notify on real data exposure, unexpected production changes or a token-spend ceiling |
| Records | For every example: date, a unique id for the input/output pair, the input, and the output or a screenshot [1] |
| Handling | Where transcripts and reproduction steps are stored, who can read them, and when they are deleted |
NIST AI 600-1 asks organisations to document the instructions given to AI red-teamers (MS-2.8-002) [5]. The same document makes the engagement repeatable: Microsoft recommends giving testers the purpose of the round, the features in scope and how to access them, the kinds of issues to look for, the time to spend, how to record results and whom to contact [1].
With scope, threat model and rules agreed, testing can start. The next lesson covers the hardest part of most scopes: agents that read untrusted content and act on it.
Try it on ThreatDeck
- Playbook: red-team an LLM application Copy its checklist into your rules of engagement and tick off scope items per system.
- Coverage Record which OWASP and ATLAS techniques each system has been tested against.
- Excessive Agency (LLM06) Use it to scope tool permissions: each tool is a potential impact.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
Impact comes from the application around the model. Testing only the endpoint can show the model is steerable but not that a steered model can reach data or take actions, which is what the owner needs to know.
-
A unique marker arriving at an attacker-controlled sink proves the path without exposing real people's data, and it keeps the engagement inside data-protection rules.
-
Agent actions leave the system. Restricting recipients to owned addresses or a test tenant keeps a successful attack from harming people outside the authorisation.
-
Those fields make a result reproducible and auditable. Severity comes later in triage, and internal model state is rarely available to an application tester.
References
- Planning red teaming for large language models (LLMs) and their applications. Microsoft Learn, 2026-05.
- GenAI Red Teaming Guide. OWASP GenAI Security Project, 2025-01.
- LLM06:2025 Excessive Agency. OWASP GenAI Security Project, 2024-11.
- Penetration Testing policy. Amazon Web Services.
- NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.