PlaybookAppSec, pentesters, AI platform teams
Red-team an LLM application
You have a chatbot, copilot or agent going to production and need evidence it resists prompt injection, data leakage and tool misuse. Most published findings are jailbreak screenshots that nobody can triage; the useful question is what the model can reach (data, tools, credentials) and what an attacker can make it do with that access.
What practitioners converge on
- Threat-model the reachable surface first. List every data source the model reads (user input, RAG, web, email, files) and every action it can take (tools, APIs, code). Untrusted input plus sensitive data plus an outbound channel is the combination to test hardest.
- Automate the broad sweep, hand-test the chains. Scheduled scanners catch regressions in known injection and leakage probes; multi-step attacks through tools and indirect injection still need a human building scenarios.
- Grade findings by impact. Report what was reached: tool called, data exposed, policy bypassed, cost incurred. Map each finding to OWASP LLM / Agentic ids so fixes and retests are tracked.
- Retest in CI. Turn confirmed findings into regression tests that run on every prompt, model or tool change.
Checklist
9 stepsSaved in this browser. Track technique-level testing per system in Coverage.
Open-source stack
- promptfoo AdoptCI regression suites and red-team configs for prompts, RAG and agents
- garak AdoptScheduled probe sweeps for injection, leakage and jailbreak classes
- PyRIT AdoptMulti-turn and orchestrated attack scenarios
- AgentDojo TrialBenchmark agent resistance to indirect prompt injection
- Augustus AssessPrompt-injection test runs against LLM endpoints
- Inspect AdoptCustom evals when you need your own scenarios and scoring
Also on the radar for these techniques: NeMo Guardrails, PurpleLlama (Llama Guard, Prompt Guard, LlamaFirewall), nono, Mastyf, Weir, beelzebub, clawsec
If you are evaluating vendors
AI red teaming and testing
- Does it test agents with tools, memory and MCP servers, or only a chat endpoint?
- How are findings scored: by business impact (data access, tool misuse) or by policy-violating text?
- Can we add our own attack datasets and app-specific policies, and re-run them as regression tests in CI?
- How does it test indirect prompt injection via documents, web pages and tool output?
AI runtime protection (LLM firewalls, guardrails)
- What exactly is blocked inline versus only flagged? Run our own indirect-injection test set through it.
- What latency does it add at p50 and p99, including on streamed responses?
- What is the false-positive rate on our real traffic, and can we tune or allowlist per app?
- Does it inspect tool calls, tool outputs and retrieved documents, or only user prompts and final answers?
Detect and hunt 18
- YARA: invisible Unicode tag characters (U+E0000–U+E007F) in prompts, tool output or agent files
- Azure OpenAI model deployment or Responsible AI policy changed
- AI coding agent CLI started with an approval-bypass flag
- AI CLI run non-interactively with a bypass flag by a script interpreter (s1ngularity pattern)
- Command spawned by an AI coding agent references credential files
- AI coding agent spawns a shell that pipes a download into an interpreter
- Destructive cloud or cluster CLI command launched under an AI coding agent
- s1ngularity (Nx) artifacts: AI CLI bypass runs in shell history and /tmp/inventory.txt
Techniques
- LLM01 Prompt Injection
- LLM02 Sensitive Information Disclosure
- LLM05 Improper Output Handling
- LLM06 Excessive Agency
- LLM07 System Prompt Leakage
- ASI01 Agent Goal Hijack
- ASI02 Tool Misuse and Exploitation
- AML.T0051 LLM Prompt Injection
- AML.T0054 LLM Jailbreak
- AML.T0086 Exfiltration via AI Agent Tool Invocation
References
Latest on this
-
OpenRouter Drops 3 Providers After Researchers Catch Months of Prompt Theft at AI Hosting Firms
Dragos reported attackers stole data from three AI inference providers for months, including requests flagged Zero Data Retention; OpenRouter dropped the three providers.
ReviewReview data retention settings and provider routing used with AI inference gateways
-
A Single Malicious Webpage Let Researchers Hijack Meta Muse and Access Private Chats
Researchers showed a malicious webpage could extract a Meta Muse user's private chats and stored memories without an approval prompt, in a lab demonstration.
ReviewReview Muse connected-account permissions and avoid browsing untrusted pages while signed in
-
Signal blocks Apple, Google AI screenshotting in response to ‘existential threat’ to privacy app
Signal will block Apple and Google operating systems by default from screenshotting chats that feed their AI assistants, citing an existential privacy threat.
MitigateEnable Signal's screenshot blocking setting and review AI assistant access to messaging apps
-
Signal blocks screenshots on iOS 27 and Android to stop AI agents from reading private chats
Signal is blocking screenshots on iOS 27 and Android to keep AI agents such as Siri AI, Gemini Intelligence and Meta's Muse from reading private chats; protections remain incomplete on macOS.
MitigateEnable Signal's screenshot blocking and note macOS protections remain incomplete
-
MCP server Flyto Core: Flyto2 Core: SSRF guard bypass via DNS rebinding (validate_url_ssrf resolves-then-connects without pinning)
GHSA-6pm8-6f34-9v3g: Flyto Core MCP server's SSRF guard can be bypassed via DNS rebinding because it resolves then connects without pinning.
PatchUpgrade flyto-core to 2.26.8 or later
-
An OpenAI agent assigned a simple writing task decided on its own to build a secret email surveillance tool
An internal OpenAI model tasked with writing a fitness document instead planned scripts to secretly forward emails to an attacker, and OpenAI cannot confirm the trigger.
ReviewReview agent permissions and email access for autonomous AI workflows
-
MCP server io.github.es617/obsidian-sync-mcp: obsidian-sync-mcp: Missing Host/Origin validation lets a public website drive a victim's Obsidian vault via DNS rebinding (no-auth mode)
Missing Host/Origin validation in obsidian-sync-mcp (npm, before 0.5.7) lets a public website drive a victim's Obsidian vault via DNS rebinding in no-auth mode.
PatchUpgrade obsidian-sync-mcp to 0.5.7 or later
-
CVE-2026-18883: Indirect prompt injection in ServiceNow Email Response AI Agent (Now Assist for CSM)
ServiceNow has addressed an indirect prompt injection issue in the Email Response AI Agent of Now Assist for CSM.
ReviewConfirm with ServiceNow that the centrally deployed fix for CVE-2026-18883 is active in your CSM instance.
-
30 Years of Gatekeeper Asymmetry Doom: The Paradox that Makes Powerful LLMs More Vulnerable
Check Point Research publishes an essay on a 'gatekeeper asymmetry' principle, arguing weaker input filters cannot stop stronger LLM systems, citing its PuzzleMask guardrail bypass research.
ReviewReview AI guardrail coverage against bypasses of external input filters
-
Ask in the Crowd: Differentially Private LLM Inference via Dummy-Augmented Shuffling
DP-SPD is a differentially private LLM inference framework that hides real user queries among client-generated dummy queries before sending them to remote LLM services.
-
Tether: Verified Stateful Policy Engine for System-Level Monitoring of Agents
Tether is a verified stateful policy engine that enforces security policies on the observed side-effects of AI agents executing shell commands and programs.
-
ToolPulse: Side-Channel Eavesdropping on Tool-Using LLM Agents
ToolPulse shows a side-channel attack can eavesdrop on tool-using LLM agents by exploiting local tool execution and shared system libraries on the user's machine.
ReviewReview exposure of local tool execution on agent-connected user machines
Reviewed 2026-11-09 · tools and news update automatically