threatdeck
Live 13:45 UTC ·

Learn / AI security fundamentals

Lesson 1 of 59 minReviewed

How LLM applications and agents become attack surface

Map the parts of an LLM application, see where untrusted text enters, and explain why the trust boundary sits in your code rather than in the model.

An LLM application is ordinary software wrapped around a model API. The model receives one block of text, the context window, and predicts what comes next. Everything the application puts into that window arrives as tokens in the same stream: the system prompt, the user's message, the conversation so far, retrieved documents, tool descriptions and tool results. The UK NCSC puts the consequence plainly: current models do not enforce a security boundary between instructions and data inside a prompt [1].

Anatomy of an LLM application and its trust boundary A user sends a prompt to an application orchestrator inside a trust boundary. The orchestrator sends the full context to a model API, receives retrieved content from RAG, files, mail and the web, and calls tools that can send, write or execute. User input and retrieved content are untrusted; only deterministic checks inside the application enforce policy. Trust boundary: your code User prompts, uploads Orchestrator system prompt · history · tool loop builds one context window Deterministic checks: authorisation, allowlists, approvals, output encoding Model (LLM API) cannot tell data from instructions Retrieved content RAG · files · mail · web pages Tools and APIs send · write · execute · pay untrusted untrusted
One context window, many writers. Only the deterministic checks inside your code enforce policy.

Where input enters

Who can put text in front of the model
ComponentWhat it holdsWho can write to it
System promptDeveloper instructions, persona, rulesYour developers. Assume it will leak: OWASP lists system prompt leakage as LLM07 [2]
User messageThe requestAnyone who can reach the interface
Retrieved contentRAG chunks, files, email, web pagesAnyone who can get text into a source the app reads, often outsiders
Tool definitionsNames, descriptions, parameter schemasWhoever wrote the tool or MCP server
Tool resultsAPI responses, command outputThe systems the tool talks to
MemoryNotes kept across sessionsEarlier sessions, including ones an attacker influenced

The third row is the one teams miss. In 2023 Greshake and colleagues showed that instructions planted in data an application retrieves, which they named indirect prompt injection, could take over real LLM-integrated applications without the attacker ever talking to the model directly [3]. The person using the app is not the attacker; the attacker is whoever wrote the web page, email or document the app read on the user's behalf.

Capability decides impact

The same injected sentence has very different consequences depending on what the application can do with the model's output:

  • Chatbot. Text goes back to the person who asked. The realistic harms are wrong or offensive answers and disclosure of the system prompt.
  • RAG application. A data path is added. The model now sees content with its own authors and access controls, so it can repeat another user's data or follow instructions hidden in a document.
  • Agent. The model chooses tools and arguments in a loop. Injected text can now send mail, change records, run commands or move money.

OWASP calls the root problem of the third case Excessive Agency (LLM06): too much functionality, too many permissions or too much autonomy granted to an LLM-based system [4]. The OWASP Top 10 for Agentic Applications (2026) catalogues the agent-specific failures that follow, such as Agent Goal Hijack (ASI01) and Tool Misuse and Exploitation (ASI02) [5].

Where the trust boundary actually is

The boundary is not the model, and it is not the system prompt. Telling a model to ignore instructions in documents reduces some attacks but does not stop a determined attacker, because the instruction and the attack arrive through the same channel. Treat the model as an untrusted component that anyone whose text reaches its context can steer, and enforce rules in deterministic code around it:

  1. Authorise every tool call with the end user's identity and permissions, not the agent's service account.
  2. Allowlist destinations for anything that leaves the system: URLs, email recipients, repositories.
  3. Require a human approval step for irreversible or high-impact actions.
  4. Encode or sanitise model output before rendering it or passing it to another interpreter.

This is the NCSC's advice: assume the model is confusable and put deterministic, non-LLM safeguards in front of anything it can do, especially tool and API use [1]. NIST's Generative AI Profile treats information security as one of twelve risks that generative AI creates or makes worse, and maps controls for it onto the AI RMF functions Govern, Map, Measure and Manage [6].

The next lesson turns this into a quick test you can apply to any design: which combinations of data access, untrusted input and outbound action make an injection exploitable.

Try it on ThreatDeck

  • Prompt Injection (LLM01) Read the technique page and the recent stories that show it in production systems.
  • Technique library Browse the OWASP LLM, OWASP Agentic and MITRE ATLAS entries this hub links to.
  • Labs & practice Try a hosted lab such as PortSwigger's Web LLM attacks to see indirect injection against a real app.

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. Why can prompt injection not be fixed the way SQL injection was fixed with parameterised queries?
  2. In an email assistant that summarises a user's inbox, whose text is most likely to carry an indirect prompt injection?
  3. Where should authorisation for an agent's tool calls be enforced?
  4. A RAG assistant gains a tool that can send email. What changes most?

References

  1. Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.
  2. LLM07:2025 System Prompt Leakage. OWASP GenAI Security Project, 2024-11.
  3. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Greshake et al. (arXiv:2302.12173), 2023-02.
  4. LLM06:2025 Excessive Agency. OWASP GenAI Security Project, 2024-11.
  5. OWASP Top 10 for Agentic Applications for 2026. OWASP GenAI Security Project, 2025-12.
  6. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.