Learn / AI security fundamentals
Lesson 1 of 59 minReviewed
How LLM applications and agents become attack surface
Map the parts of an LLM application, see where untrusted text enters, and explain why the trust boundary sits in your code rather than in the model.
An LLM application is ordinary software wrapped around a model API. The model receives one block of text, the context window, and predicts what comes next. Everything the application puts into that window arrives as tokens in the same stream: the system prompt, the user's message, the conversation so far, retrieved documents, tool descriptions and tool results. The UK NCSC puts the consequence plainly: current models do not enforce a security boundary between instructions and data inside a prompt [1].
Where input enters
| Component | What it holds | Who can write to it |
|---|---|---|
| System prompt | Developer instructions, persona, rules | Your developers. Assume it will leak: OWASP lists system prompt leakage as LLM07 [2] |
| User message | The request | Anyone who can reach the interface |
| Retrieved content | RAG chunks, files, email, web pages | Anyone who can get text into a source the app reads, often outsiders |
| Tool definitions | Names, descriptions, parameter schemas | Whoever wrote the tool or MCP server |
| Tool results | API responses, command output | The systems the tool talks to |
| Memory | Notes kept across sessions | Earlier sessions, including ones an attacker influenced |
The third row is the one teams miss. In 2023 Greshake and colleagues showed that instructions planted in data an application retrieves, which they named indirect prompt injection, could take over real LLM-integrated applications without the attacker ever talking to the model directly [3]. The person using the app is not the attacker; the attacker is whoever wrote the web page, email or document the app read on the user's behalf.
Capability decides impact
The same injected sentence has very different consequences depending on what the application can do with the model's output:
- Chatbot. Text goes back to the person who asked. The realistic harms are wrong or offensive answers and disclosure of the system prompt.
- RAG application. A data path is added. The model now sees content with its own authors and access controls, so it can repeat another user's data or follow instructions hidden in a document.
- Agent. The model chooses tools and arguments in a loop. Injected text can now send mail, change records, run commands or move money.
OWASP calls the root problem of the third case Excessive Agency (LLM06): too much functionality, too many permissions or too much autonomy granted to an LLM-based system [4]. The OWASP Top 10 for Agentic Applications (2026) catalogues the agent-specific failures that follow, such as Agent Goal Hijack (ASI01) and Tool Misuse and Exploitation (ASI02) [5].
Where the trust boundary actually is
The boundary is not the model, and it is not the system prompt. Telling a model to ignore instructions in documents reduces some attacks but does not stop a determined attacker, because the instruction and the attack arrive through the same channel. Treat the model as an untrusted component that anyone whose text reaches its context can steer, and enforce rules in deterministic code around it:
- Authorise every tool call with the end user's identity and permissions, not the agent's service account.
- Allowlist destinations for anything that leaves the system: URLs, email recipients, repositories.
- Require a human approval step for irreversible or high-impact actions.
- Encode or sanitise model output before rendering it or passing it to another interpreter.
This is the NCSC's advice: assume the model is confusable and put deterministic, non-LLM safeguards in front of anything it can do, especially tool and API use [1]. NIST's Generative AI Profile treats information security as one of twelve risks that generative AI creates or makes worse, and maps controls for it onto the AI RMF functions Govern, Map, Measure and Manage [6].
The next lesson turns this into a quick test you can apply to any design: which combinations of data access, untrusted input and outbound action make an injection exploitable.
Try it on ThreatDeck
- Prompt Injection (LLM01) Read the technique page and the recent stories that show it in production systems.
- Technique library Browse the OWASP LLM, OWASP Agentic and MITRE ATLAS entries this hub links to.
- Labs & practice Try a hosted lab such as PortSwigger's Web LLM attacks to see indirect injection against a real app.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
Parameterised queries work because the database parses code and data separately. A model receives one token stream and has no equivalent boundary, which is why the NCSC calls it a confusable deputy and recommends limiting impact instead of expecting a complete fix.
-
Indirect injection arrives through retrieved content. Incoming mail is written by anyone who can email the user, so it is the main untrusted input even though the user is not malicious.
-
Instructions in the prompt can be overridden by injected text that arrives through the same channel. Checks in code, using the identity of the person the agent acts for, hold no matter what the model was persuaded to do.
-
Capability decides impact. The same injected document that could only distort an answer can now trigger an outbound action, which is the Excessive Agency (LLM06) problem.
References
- Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.
- LLM07:2025 System Prompt Leakage. OWASP GenAI Security Project, 2024-11.
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Greshake et al. (arXiv:2302.12173), 2023-02.
- LLM06:2025 Excessive Agency. OWASP GenAI Security Project, 2024-11.
- OWASP Top 10 for Agentic Applications for 2026. OWASP GenAI Security Project, 2025-12.
- NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.