threatdeck
Live 13:45 UTC ·

Learn / Red teaming AI systems

Lesson 5 of 68 minReviewed

Impact-based findings and reporting

Rate AI findings by what the attacker achieved, make nondeterministic results reproducible, map them to OWASP and ATLAS ids, and hand over a fix plus a regression test.

Security practitioners are sceptical of AI findings for a good reason: many reports show a model saying something odd and stop there. A finding earns attention when it shows what an attacker in a stated position actually achieved, with evidence, and what will stop it.

Severity follows impact

Microsoft's published severity classification for AI systems is a useful anchor. For prompt injection it rates by consequence: injection that leads to data exfiltration or privileged actions affecting another user receives a security severity, higher when no user interaction is needed, while injection that only influences the generated output is routed to content-related issues instead [1]. Apply the same logic to your own triage:

Same technique, different findings
What the tester showedAttacker positionReasonable treatment
Model reveals its system promptSigned-in userLow, unless the prompt holds secrets or authorisation logic (which is itself the finding)
Injected document changes a summary's wordingAnyone who can share a documentContent or quality issue; track, do not page anyone
Injected email makes the agent send a canary file to an outside addressAnyone who can email the user, no clicksHigh: cross-user disclosure with zero interaction
Injected ticket makes the agent run a shell command in CIAnyone who can open a ticketCritical if the runner holds credentials

Make nondeterminism reproducible

A result that appeared once cannot be triaged. Record what makes it repeatable: the date, a unique id for the input/output pair, the exact input and the output or a screenshot [2], plus the model version, sampling settings, tool configuration and retrieval state. Then rerun it and report successes out of attempts. NIST's agent evaluations showed why: repeated attempts changed per-task success rates substantially, and aggregate rates hid which tasks carried serious consequences [3].

Minimal finding record
id: RT-2026-014
title: Inbound email causes agent to send files to an external address
attacker: anyone who can email a user (no interaction)
impact: cross-user disclosure; canary file received at tester-owned mailbox
rate: 7 of 20 attempts (model and version, temperature 0.7, tools as deployed)
maps_to: LLM01, ASI02, AML.T0051, AML.T0086
fix: allowlist recipients in the send tool; require approval for attachments
regression: case RT-2026-014 added to CI suite, must fail closed

Map to shared ids

Tag each finding with the framework ids it demonstrates, such as LLM01 for the injection, ASI02 for the tool misuse and the matching ATLAS techniques [4, 7, 5]. Ids let you count findings per class across systems, link them to detections and coverage, and avoid arguing about names. Keep the reproduction steps themselves in your private tracker: a report for executives or a public write-up needs the impact and the fix, not a working recipe.

Recommend a fix that does not depend on the model

"Improve the system prompt" is not a fix. Prefer controls that hold whatever the model is persuaded to do: allowlisted destinations, authorisation with the end user's identity, approval for irreversible actions, output encoding, and removal of tools the task does not need. This is the NCSC's guidance for a confusable model: deterministic safeguards around what it can do, rather than reliance on the model to refuse [6]. Every finding should leave behind a regression case that fails before the fix and passes after it.

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. Which finding deserves the highest severity?
  2. A tester saw an exploit work once and could not repeat it. What should the finding record include before triage?
  3. Which recommendation is the strongest fix for an exfiltration-by-email finding?
  4. Why keep step-by-step reproduction details out of an executive summary or public write-up?

References

  1. Microsoft Vulnerability Severity Classification for AI Systems. Microsoft Security Response Center.
  2. Planning red teaming for large language models (LLMs) and their applications. Microsoft Learn, 2026-05.
  3. Technical Blog: Strengthening AI Agent Hijacking Evaluations. NIST (US AI Safety Institute), 2025-01.
  4. OWASP Top 10 for LLM Applications 2025. OWASP GenAI Security Project, 2024-11.
  5. MITRE ATLAS (Adversarial Threat Landscape for AI Systems). MITRE.
  6. Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.
  7. OWASP Top 10 for Agentic Applications for 2026. OWASP GenAI Security Project, 2025-12.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.