Learn / Red teaming AI systems
Lesson 3 of 69 minReviewed
Testing agents and indirect prompt injection
Design agent tests where the attacker controls a tool output and the goal is an action, measure utility and attack success separately, and avoid the three ways agent results mislead: single attempts, aggregates and static attacks.
Testing a chatbot asks whether the model can be made to say something. Testing an agent asks whether a third party can make it do something. The attacker usually never talks to the agent: they place text where the agent will read it while working for a legitimate user, the indirect prompt injection pattern Greshake and colleagues described in 2023 [4]. OWASP lists the agent outcome as Agent Goal Hijack (ASI01) [5].
How AgentDojo structures the problem
AgentDojo, from ETH Zurich, is a useful model for designing your own tests. It contains 97 realistic user tasks, such as managing an email client, using an e-banking site or booking travel, and 629 security test cases across four environments: Workspace, Slack, Travel and Banking [1]. Each security case pairs a user task with an injection task, and the injection is placed in data a tool returns, for example an email in the inbox or a web page the agent reads [1]. The authors describe it as an extensible environment rather than a static test suite, so new tasks, defences and adaptive attacks can be added [1].
| Metric | Question it answers |
|---|---|
| Benign utility | With no attack, how many user tasks does the agent complete? |
| Utility under attack | With an injection present, does the agent still complete the user task without the attacker's side effects? |
| Targeted attack success rate | How often does the agent carry out the attacker's chosen action? |
Keep them apart in your own reports. A defence that drives attack success to zero by making the agent useless has not solved anything, and a high benign utility says nothing about security.
Three ways agent results mislead
NIST's Center for AI Standards and Innovation ran AgentDojo against an agent built on a model that was among the most resistant to hijacking, and published what changed its conclusions [2]:
- Static attacks understate risk. Red teaming with the UK AI Security Institute raised attack success on held-out Workspace tasks from 11% for the strongest baseline attack to 81% for the strongest new attack [2]. Independent work bypassed 12 published defences with success above 90% for most, many of which had reported near-zero rates against fixed attacks [3].
- Single attempts understate risk. Outputs are probabilistic. Running each of five injection tasks 25 times raised the average success rate from 57% to 80% [2]. An attacker who can retry, for example by sending many emails, gets the higher number.
- Aggregates hide the dangerous case. The 57% average covered tasks with very different rates and consequences; a data-exfiltration task that succeeded less often could still matter most [2].
Design your own agent tests
- Use the agent's real tools pointed at test tenants, and put the attacker's text where an outsider could put it: an inbound email, a shared document, a ticket, a web page.
- Define the attacker goal as an observable action: a canary string reaching an address you own, a record changed, a command run in a sandbox.
- Run each case several times and report successes out of attempts with the model version and settings.
- Include multi-turn cases. Escalation across turns behaves differently from one message; PyRIT, for example, ships multi-turn strategies alongside single-turn ones [6].
- When a defence is added, change the attack. Measuring a filter only against the inputs it was tuned on repeats the mistake the adaptive-attack results describe [3].
Try it on ThreatDeck
- Agent Goal Hijack (ASI01) Read the agentic risk these tests target and the stories where it happened.
- AgentDojo on the radar Check the benchmark's status and related tools before adapting its task design.
- Labs & practice Practise on a deliberately vulnerable agent or MCP server in an isolated environment.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
Indirect injection reaches the agent through tool outputs. The user is not the attacker, which is why the test pairs an ordinary user task with an injection placed in content the agent reads.
-
Security and utility are separate measurements. A defence that makes the agent fail most legitimate tasks will be switched off in practice, so both numbers belong in the result.
-
Repeated attempts raised the average from 57% to 80%. When retries are cheap for an attacker, the repeated-attempt rate is the realistic one.
-
Adaptive attackers change strategy once a defence is known. Work evaluating 12 defences found most were bypassed above 90% under adaptive attack despite near-zero reported rates.
References
- AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents. Debenedetti et al., ETH Zurich (arXiv:2406.13352), 2024-06.
- Technical Blog: Strengthening AI Agent Hijacking Evaluations. NIST (US AI Safety Institute), 2025-01.
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses against LLM Jailbreaks and Prompt Injections. Nasr et al. (arXiv:2510.09023), 2025-10.
- Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection. Greshake et al. (arXiv:2302.12173), 2023-02.
- OWASP Top 10 for Agentic Applications for 2026. OWASP GenAI Security Project, 2025-12.
- PyRIT documentation. Microsoft.