threatdeck
Live 13:45 UTC ·

Learn / Securing LLM applications

Lesson 1 of 610 minReviewed

Prompt injection: direct, indirect, and the designs that contain it

Tell direct from indirect prompt injection, know which defences are probabilistic, and choose an architecture pattern that keeps injected text from triggering consequential actions.

The term dates from September 2022: Riley Goodside showed GPT-3 following instructions hidden in its input, and Simon Willison named the problem prompt injection by analogy with SQL injection, because both arise when untrusted input is concatenated into instructions [1]. OWASP lists it first in the 2025 Top 10 for LLM applications and covers two variants [3].

Two ways injected instructions reach the model
DirectIndirect
AttackerThe person typing into the appWhoever wrote content the app reads for someone else
Entry pointUser messageWeb pages, email, documents, tool output, RAG chunks
Typical goalBypass rules, extract the system promptExfiltrate the victim's data, act on the victim's behalf
Who is harmedMostly the operatorThe user, who never sees the attack text

Greshake and colleagues showed the indirect form against real applications in 2023 [2]. It is the variant that matters most for agents, because the victim does nothing wrong: they ask for a summary and the document does the rest.

Layers that reduce the odds

Microsoft's published approach starts with probabilistic layers [4]. Hardened system prompts tell the model how to treat external text. Spotlighting marks untrusted content so the model can tell where it came from, using random delimiters, a data marker inserted throughout the text, or an encoding such as base64 [4]. In the original paper, spotlighting cut attack success on GPT-family models from above 50% to below 2% in the authors' tests [5]. Classifiers such as Prompt Shields then flag injection attempts in user input and documents [4].

Microsoft states that its design does not rely on blocking every injection; it aims to make a successful injection harmless through deterministic controls and user approval [4]. That is the right mental model: these layers lower the rate, architecture decides the impact.

Patterns that contain the impact

Beurer-Kellner and co-authors describe six patterns built on one principle: once an agent has read untrusted input, it must be constrained so that the input cannot trigger consequential actions [6].

Design patterns for prompt-injection resistance (after Beurer-Kellner et al.)
PatternHow it constrains the modelCost
Action-selectorMaps a request to one of a fixed set of actions and never sees their resultsNo open-ended tasks
Plan-then-executeCommits to the sequence of tool calls before reading untrusted dataData can still shape arguments
LLM map-reduceIsolated instances each handle one untrusted item without dangerous toolsExtra calls, constrained outputs
Dual LLMA privileged model with tools sees only references to text a quarantined model readComplexity, weaker UX
Code-then-executeThe model writes a program; the program, not the model, touches untrusted dataNeeds a safe interpreter
Context-minimisationRemoves earlier content from context once it has served its purposeLess conversational memory
Dual LLM pattern: privileged and quarantined models with a plain-code controller The user talks to a controller written in ordinary code. The controller asks a privileged model to plan and choose tools; that model only ever sees variable names such as VAR1, never untrusted text. A quarantined model with no tool access reads the untrusted email, web or document content and returns results that the controller stores as variables. Only the controller calls tools, with arguments it checks. Untrusted content and the quarantined model form a tainted zone whose output never reaches the privileged model directly. Tainted zone: its raw output never reaches the privileged LLM User trusted request Privileged LLM plans, picks tools; sees $VAR1 only Controller (plain code) stores tainted values as $VAR1… checks every tool argument Quarantined LLM reads untrusted text; no tools Tools called only by the controller with allowlisted arguments Untrusted content email · web pages · documents untrusted
Dual LLM: plain code holds tainted values as variables, so the model that can call tools never reads them.

Willison proposed the dual LLM pattern in 2023 and was candid that it adds complexity and is not complete [7]. CaMeL, from Debenedetti and colleagues, develops the idea: it derives control and data flow from the trusted user query, attaches capabilities to every value, and enforces policy when tools are called, so retrieved data cannot change what the program does. It solved 77% of AgentDojo tasks with provable security, against 84% for an undefended agent [8].

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. A recruiter's assistant summarises CVs. One CV contains hidden text telling the assistant to rank it first. What kind of attack is this?
  2. Which statement about spotlighting is accurate?
  3. In plan-then-execute, what can injected data in a retrieved email still influence?
  4. In the dual LLM pattern, which component is allowed to call tools?

References

  1. Prompt injection attacks against GPT-3. Simon Willison, 2022-09-12.
  2. Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Greshake et al. (arXiv:2302.12173), 2023-02.
  3. LLM01:2025 Prompt Injection. OWASP GenAI Security Project, 2024-11.
  4. How Microsoft defends against indirect prompt injection attacks. Microsoft Security Response Center, 2025-07-29.
  5. Defending Against Indirect Prompt Injection Attacks With Spotlighting. Hines et al. (arXiv:2403.14720), 2024-03.
  6. Design Patterns for Securing LLM Agents against Prompt Injections. Beurer-Kellner et al. (arXiv:2506.08837), 2025-06.
  7. The Dual LLM pattern for building AI assistants that can resist prompt injection. Simon Willison, 2023-04-25.
  8. Defeating Prompt Injections by Design (CaMeL). Debenedetti et al. (arXiv:2503.18813), 2025-03.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.