threatdeck
Live 13:45 UTC ·

Learn / AI security fundamentals

Lesson 5 of 510 minReviewed

Threat-model an AI feature in an hour

Run a short, repeatable threat model for an LLM feature or agent: data flow, trust boundaries, the Rule of Two test, framework ids, controls, tests and logs.

The UK NCSC's guidelines for secure AI system development split the work into secure design, development, deployment, and operation and maintenance, and put threat modelling in the design stage [1]. In NIST's AI Risk Management Framework the same activity sits under the Map function, one of four core functions alongside Govern, Measure and Manage [2]. You do not need a heavyweight process to get value from it. The steps below fit in an hour with the feature owner and one security engineer.

1. Sketch the flow (10 minutes)

Draw the components from lesson 1: user, orchestrator, model, each retrieval source, each tool or MCP server, and memory. For every arrow, write who controls the text on it. Include the paths people forget: tool descriptions, error messages returned to the model, files the agent can open, and any server a user can add later.

2. Run the Rule of Two (5 minutes)

Mark each source or tool as untrusted input (A), sensitive data or system (B), or state change and outbound communication (C). If one session holds all three, the design needs a change or a human approval before C actions, as Meta's guidance states [3]. Record which leg you removed; that decision is the most important line in the document.

3. Fill the threat table (25 minutes)

Worked example: an internal docs assistant that can open tickets
ThreatIdsControl (deterministic)TestLog
Instructions hidden in a wiki page make the assistant file tickets with internal data in a public projectLLM01, ASI01, AML.T0051Ticket tool restricted to internal projects; user confirms ticket bodyIndirect-injection case in CI using a seeded wiki pageTool calls with arguments, source documents in context
Retrieval returns pages the asking user cannot readLLM08Filter results by the user's permissions at query timeTwo test users with different access ask the same questionRetrieved document ids per answer
Rendered answer loads an attacker image URL carrying dataLLM05Strip external images and links not on an allowlist before renderingOutput containing an external image is blockedBlocked render events
Ticket API token is broader than neededASI03, LLM06Scoped token, write access to the ticket queue onlyToken cannot read other systemsAPI audit log for the token

Keep each row concrete: who does what, through which path, with what effect. Controls must be enforced in code or configuration, not by an instruction to the model. OWASP's Agentic AI Threats and Mitigations guide gives a fuller threat catalogue to check your rows against [4], and the OWASP AI Agent Security Cheat Sheet groups controls by area, from tool least privilege to monitoring [5].

4. Decide what to measure (10 minutes)

Every row needs a test that would fail if the control broke, and a log that would show the threat being attempted in production. The NCSC recommends logging enough to spot suspicious activity, potentially including full model inputs, outputs and tool calls, and notes that failed tool calls can reveal an attacker probing [6]. Decide now where those logs go and who reviews them; prompts and outputs may contain personal data, so set retention deliberately.

5. Close the loop

  • Store the table with the feature's design document and link it from the code that implements each control.
  • Re-run the hour when a tool, data source or MCP server is added, when the model changes, or after an incident.
  • Track residual risk explicitly. NIST's Generative AI Profile lists information security among the risks generative AI creates or exacerbates; some prompt-injection risk will remain whatever you do [7, 6].

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. In the NIST AI RMF, under which core function does mapping an AI system's context and risks, such as a threat model, primarily sit?
  2. Which is an acceptable control for the row "injected wiki page makes the assistant file a ticket with internal data"?
  3. Why should every threat-table row have a log column?
  4. When should the one-hour threat model be repeated?

References

  1. Guidelines for secure AI system development. UK National Cyber Security Centre, 2023-11.
  2. NIST AI 100-1: Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST, 2023-01.
  3. Agents Rule of Two: A Practical Approach to AI Agent Security. Meta AI, 2025-10-31.
  4. Agentic AI – Threats and Mitigations. OWASP GenAI Security Project, 2025-02.
  5. AI Agent Security Cheat Sheet. OWASP Cheat Sheet Series.
  6. Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.
  7. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.