Learn / AI security fundamentals
Lesson 5 of 510 minReviewed
Threat-model an AI feature in an hour
Run a short, repeatable threat model for an LLM feature or agent: data flow, trust boundaries, the Rule of Two test, framework ids, controls, tests and logs.
The UK NCSC's guidelines for secure AI system development split the work into secure design, development, deployment, and operation and maintenance, and put threat modelling in the design stage [1]. In NIST's AI Risk Management Framework the same activity sits under the Map function, one of four core functions alongside Govern, Measure and Manage [2]. You do not need a heavyweight process to get value from it. The steps below fit in an hour with the feature owner and one security engineer.
1. Sketch the flow (10 minutes)
Draw the components from lesson 1: user, orchestrator, model, each retrieval source, each tool or MCP server, and memory. For every arrow, write who controls the text on it. Include the paths people forget: tool descriptions, error messages returned to the model, files the agent can open, and any server a user can add later.
2. Run the Rule of Two (5 minutes)
Mark each source or tool as untrusted input (A), sensitive data or system (B), or state change and outbound communication (C). If one session holds all three, the design needs a change or a human approval before C actions, as Meta's guidance states [3]. Record which leg you removed; that decision is the most important line in the document.
3. Fill the threat table (25 minutes)
| Threat | Ids | Control (deterministic) | Test | Log |
|---|---|---|---|---|
| Instructions hidden in a wiki page make the assistant file tickets with internal data in a public project | LLM01, ASI01, AML.T0051 | Ticket tool restricted to internal projects; user confirms ticket body | Indirect-injection case in CI using a seeded wiki page | Tool calls with arguments, source documents in context |
| Retrieval returns pages the asking user cannot read | LLM08 | Filter results by the user's permissions at query time | Two test users with different access ask the same question | Retrieved document ids per answer |
| Rendered answer loads an attacker image URL carrying data | LLM05 | Strip external images and links not on an allowlist before rendering | Output containing an external image is blocked | Blocked render events |
| Ticket API token is broader than needed | ASI03, LLM06 | Scoped token, write access to the ticket queue only | Token cannot read other systems | API audit log for the token |
Keep each row concrete: who does what, through which path, with what effect. Controls must be enforced in code or configuration, not by an instruction to the model. OWASP's Agentic AI Threats and Mitigations guide gives a fuller threat catalogue to check your rows against [4], and the OWASP AI Agent Security Cheat Sheet groups controls by area, from tool least privilege to monitoring [5].
4. Decide what to measure (10 minutes)
Every row needs a test that would fail if the control broke, and a log that would show the threat being attempted in production. The NCSC recommends logging enough to spot suspicious activity, potentially including full model inputs, outputs and tool calls, and notes that failed tool calls can reveal an attacker probing [6]. Decide now where those logs go and who reviews them; prompts and outputs may contain personal data, so set retention deliberately.
5. Close the loop
- Store the table with the feature's design document and link it from the code that implements each control.
- Re-run the hour when a tool, data source or MCP server is added, when the model changes, or after an incident.
- Track residual risk explicitly. NIST's Generative AI Profile lists information security among the risks generative AI creates or exacerbates; some prompt-injection risk will remain whatever you do [7, 6].
Try it on ThreatDeck
- Coverage tracker Record which OWASP and ATLAS ids each AI system has been threat-modelled and tested for.
- Red-team an LLM application Turn the Test column into a structured test plan.
- Agent identity and least privilege Work through the scoped-credential controls for the ASI03 row.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
The AI RMF Core has four functions: Govern, Map, Measure and Manage. Establishing context and identifying risks is Map; testing them is Measure; treating them is Manage.
-
Controls must hold even if the model is persuaded. A code-level restriction plus a human confirmation does; a prompt instruction arrives through the same channel the attack uses.
-
Tests show the control works before release; logs show whether anyone is trying to get around it afterwards. The NCSC notes failed tool calls can be an early sign of probing.
-
Adding a tool or data source can complete the lethal trifecta in a design that was safe yesterday, so changes to capability are the trigger.
References
- Guidelines for secure AI system development. UK National Cyber Security Centre, 2023-11.
- NIST AI 100-1: Artificial Intelligence Risk Management Framework (AI RMF 1.0). NIST, 2023-01.
- Agents Rule of Two: A Practical Approach to AI Agent Security. Meta AI, 2025-10-31.
- Agentic AI – Threats and Mitigations. OWASP GenAI Security Project, 2025-02.
- AI Agent Security Cheat Sheet. OWASP Cheat Sheet Series.
- Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.
- NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.