Learn / Red teaming AI systems
Lesson 5 of 68 minReviewed
Impact-based findings and reporting
Rate AI findings by what the attacker achieved, make nondeterministic results reproducible, map them to OWASP and ATLAS ids, and hand over a fix plus a regression test.
Security practitioners are sceptical of AI findings for a good reason: many reports show a model saying something odd and stop there. A finding earns attention when it shows what an attacker in a stated position actually achieved, with evidence, and what will stop it.
Severity follows impact
Microsoft's published severity classification for AI systems is a useful anchor. For prompt injection it rates by consequence: injection that leads to data exfiltration or privileged actions affecting another user receives a security severity, higher when no user interaction is needed, while injection that only influences the generated output is routed to content-related issues instead [1]. Apply the same logic to your own triage:
| What the tester showed | Attacker position | Reasonable treatment |
|---|---|---|
| Model reveals its system prompt | Signed-in user | Low, unless the prompt holds secrets or authorisation logic (which is itself the finding) |
| Injected document changes a summary's wording | Anyone who can share a document | Content or quality issue; track, do not page anyone |
| Injected email makes the agent send a canary file to an outside address | Anyone who can email the user, no clicks | High: cross-user disclosure with zero interaction |
| Injected ticket makes the agent run a shell command in CI | Anyone who can open a ticket | Critical if the runner holds credentials |
Make nondeterminism reproducible
A result that appeared once cannot be triaged. Record what makes it repeatable: the date, a unique id for the input/output pair, the exact input and the output or a screenshot [2], plus the model version, sampling settings, tool configuration and retrieval state. Then rerun it and report successes out of attempts. NIST's agent evaluations showed why: repeated attempts changed per-task success rates substantially, and aggregate rates hid which tasks carried serious consequences [3].
id: RT-2026-014
title: Inbound email causes agent to send files to an external address
attacker: anyone who can email a user (no interaction)
impact: cross-user disclosure; canary file received at tester-owned mailbox
rate: 7 of 20 attempts (model and version, temperature 0.7, tools as deployed)
maps_to: LLM01, ASI02, AML.T0051, AML.T0086
fix: allowlist recipients in the send tool; require approval for attachments
regression: case RT-2026-014 added to CI suite, must fail closedMap to shared ids
Tag each finding with the framework ids it demonstrates, such as LLM01 for the injection, ASI02 for the tool misuse and the matching ATLAS techniques [4, 7, 5]. Ids let you count findings per class across systems, link them to detections and coverage, and avoid arguing about names. Keep the reproduction steps themselves in your private tracker: a report for executives or a public write-up needs the impact and the fix, not a working recipe.
Recommend a fix that does not depend on the model
"Improve the system prompt" is not a fix. Prefer controls that hold whatever the model is persuaded to do: allowlisted destinations, authorisation with the end user's identity, approval for irreversible actions, output encoding, and removal of tools the task does not need. This is the NCSC's guidance for a confusable model: deterministic safeguards around what it can do, rather than reliance on the model to refuse [6]. Every finding should leave behind a regression case that fails before the fix and passes after it.
Try it on ThreatDeck
- ATLAS: LLM Prompt Injection (AML.T0051) Use the id and its mitigations when tagging injection findings.
- Tool Misuse and Exploitation (ASI02) Check the agentic id for findings where injection reaches a tool.
- Coverage Record which techniques each system has findings or clean tests for.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
Severity follows what the attacker achieves and how easily. Zero-interaction cross-user disclosure crosses a trust boundary; the others are low-impact or content-quality issues.
-
Outputs are probabilistic, so reproducibility comes from recording the conditions and rerunning. A rate such as 3 of 20 is a reportable result; a single unexplained success is not yet one.
-
A deterministic control in the tool holds no matter what the model was persuaded to do. Prompt instructions arrive through the same channel as the attack, and a larger model is still confusable.
-
Engineers who fix and retest need the details in a controlled tracker. Decision makers and the public need what was possible, how bad it was and what changed.
References
- Microsoft Vulnerability Severity Classification for AI Systems. Microsoft Security Response Center.
- Planning red teaming for large language models (LLMs) and their applications. Microsoft Learn, 2026-05.
- Technical Blog: Strengthening AI Agent Hijacking Evaluations. NIST (US AI Safety Institute), 2025-01.
- OWASP Top 10 for LLM Applications 2025. OWASP GenAI Security Project, 2024-11.
- MITRE ATLAS (Adversarial Threat Landscape for AI Systems). MITRE.
- Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.
- OWASP Top 10 for Agentic Applications for 2026. OWASP GenAI Security Project, 2025-12.