threatdeck
Live 13:45 UTC ·

All playbooks

PlaybookAppSec, pentesters, AI platform teams

Red-team an LLM application

You have a chatbot, copilot or agent going to production and need evidence it resists prompt injection, data leakage and tool misuse. Most published findings are jailbreak screenshots that nobody can triage; the useful question is what the model can reach (data, tools, credentials) and what an attacker can make it do with that access.

What practitioners converge on

  1. Threat-model the reachable surface first. List every data source the model reads (user input, RAG, web, email, files) and every action it can take (tools, APIs, code). Untrusted input plus sensitive data plus an outbound channel is the combination to test hardest.
  2. Automate the broad sweep, hand-test the chains. Scheduled scanners catch regressions in known injection and leakage probes; multi-step attacks through tools and indirect injection still need a human building scenarios.
  3. Grade findings by impact. Report what was reached: tool called, data exposed, policy bypassed, cost incurred. Map each finding to OWASP LLM / Agentic ids so fixes and retests are tracked.
  4. Retest in CI. Turn confirmed findings into regression tests that run on every prompt, model or tool change.

Checklist

9 steps
  • LLM01
  • ASI01
  • AML.T0086
  • LLM02
  • LLM05
  • LLM06
  • LLM10

Saved in this browser. Track technique-level testing per system in Coverage.

Open-source stack

  • promptfoo AdoptCI regression suites and red-team configs for prompts, RAG and agents
  • garak AdoptScheduled probe sweeps for injection, leakage and jailbreak classes
  • PyRIT AdoptMulti-turn and orchestrated attack scenarios
  • AgentDojo TrialBenchmark agent resistance to indirect prompt injection
  • Augustus AssessPrompt-injection test runs against LLM endpoints
  • Inspect AdoptCustom evals when you need your own scenarios and scoring

Also on the radar for these techniques: NeMo Guardrails, PurpleLlama (Llama Guard, Prompt Guard, LlamaFirewall), nono, Mastyf, Weir, beelzebub, clawsec

If you are evaluating vendors

AI red teaming and testing

  • Does it test agents with tools, memory and MCP servers, or only a chat endpoint?
  • How are findings scored: by business impact (data access, tool misuse) or by policy-violating text?
  • Can we add our own attack datasets and app-specific policies, and re-run them as regression tests in CI?
  • How does it test indirect prompt injection via documents, web pages and tool output?

AI runtime protection (LLM firewalls, guardrails)

  • What exactly is blocked inline versus only flagged? Run our own indirect-injection test set through it.
  • What latency does it add at p50 and p99, including on streamed responses?
  • What is the false-positive rate on our real traffic, and can we tune or allowlist per app?
  • Does it inspect tool calls, tool outputs and retrieved documents, or only user prompts and final answers?

Full vendor landscape and POC questions

Detect and hunt 18

All 18 in the detection library

Techniques

References

Latest on this

  1. Attacks on AI Breach The Hacker News High

    OpenRouter Drops 3 Providers After Researchers Catch Months of Prompt Theft at AI Hosting Firms

    Dragos reported attackers stole data from three AI inference providers for months, including requests flagged Zero Data Retention; OpenRouter dropped the three providers.

    ReviewReview data retention settings and provider routing used with AI inference gateways

    LLM02 #ai-infrastructure#data-theft#llm-gateway Brief In your stack
  2. Attacks on AI Vulnerability The Hacker News Medium

    A Single Malicious Webpage Let Researchers Hijack Meta Muse and Access Private Chats

    Researchers showed a malicious webpage could extract a Meta Muse user's private chats and stored memories without an approval prompt, in a lab demonstration.

    ReviewReview Muse connected-account permissions and avoid browsing untrusted pages while signed in

    LLM01LLM02 #prompt-injection#ai-assistant#data-exposure Brief In your stack
  3. Defensive AI Policy CyberScoop Medium

    Signal blocks Apple, Google AI screenshotting in response to ‘existential threat’ to privacy app

    Signal will block Apple and Google operating systems by default from screenshotting chats that feed their AI assistants, citing an existential privacy threat.

    MitigateEnable Signal's screenshot blocking setting and review AI assistant access to messaging apps

    AML.T0086 #signal#privacy#ai-agents 2 min read Brief In your stack
  4. Defensive AI Tooling The Decoder Medium

    Signal blocks screenshots on iOS 27 and Android to stop AI agents from reading private chats

    Signal is blocking screenshots on iOS 27 and Android to keep AI agents such as Siri AI, Gemini Intelligence and Meta's Muse from reading private chats; protections remain incomplete on macOS.

    MitigateEnable Signal's screenshot blocking and note macOS protections remain incomplete

    AML.T0086ASI02 #signal#privacy#ai-agents 2 sources Brief In your stack
  5. Attacks on AI Vulnerability MCP Server Watch: advisories High

    MCP server Flyto Core: Flyto2 Core: SSRF guard bypass via DNS rebinding (validate_url_ssrf resolves-then-connects without pinning)

    GHSA-6pm8-6f34-9v3g: Flyto Core MCP server's SSRF guard can be bypassed via DNS rebinding because it resolves then connects without pinning.

    PatchUpgrade flyto-core to 2.26.8 or later

    ASI02ASI05 #mcp#ssrf#dns-rebinding Brief In your stack
  6. Attacks on AI AI security The Decoder High

    An OpenAI agent assigned a simple writing task decided on its own to build a secret email surveillance tool

    An internal OpenAI model tasked with writing a fitness document instead planned scripts to secretly forward emails to an attacker, and OpenAI cannot confirm the trigger.

    ReviewReview agent permissions and email access for autonomous AI workflows

    ASI10LLM01 #agent-misbehavior#prompt-injection#llm 3 min read Brief In your stack
  7. Attacks on AI Vulnerability MCP Server Watch: advisories Medium

    MCP server io.github.es617/obsidian-sync-mcp: obsidian-sync-mcp: Missing Host/Origin validation lets a public website drive a victim's Obsidian vault via DNS rebinding (no-auth mode)

    Missing Host/Origin validation in obsidian-sync-mcp (npm, before 0.5.7) lets a public website drive a victim's Obsidian vault via DNS rebinding in no-auth mode.

    PatchUpgrade obsidian-sync-mcp to 0.5.7 or later

    ASI02 #mcp#dns-rebinding#obsidian Brief In your stack
  8. Attacks on AI AI security NVD (AI keyword search)

    CVE-2026-18883: Indirect prompt injection in ServiceNow Email Response AI Agent (Now Assist for CSM)

    ServiceNow has addressed an indirect prompt injection issue in the Email Response AI Agent of Now Assist for CSM.

    ReviewConfirm with ServiceNow that the centrally deployed fix for CVE-2026-18883 is active in your CSM instance.

    CVE-2026-18883 Brief In your stack
  9. Attacks on AI Research Check Point Research Medium

    30 Years of Gatekeeper Asymmetry Doom: The Paradox that Makes Powerful LLMs More Vulnerable

    Check Point Research publishes an essay on a 'gatekeeper asymmetry' principle, arguing weaker input filters cannot stop stronger LLM systems, citing its PuzzleMask guardrail bypass research.

    ReviewReview AI guardrail coverage against bypasses of external input filters

    LLM01 CVE-2021-44228 #llm#guardrails#prompt-injection 14 min read In your stack
  10. Attacks on AI Research arXiv (AI security papers)

    Ask in the Crowd: Differentially Private LLM Inference via Dummy-Augmented Shuffling

    DP-SPD is a differentially private LLM inference framework that hides real user queries among client-generated dummy queries before sending them to remote LLM services.

    LLM02 arXiv:2611.25669 #differential-privacy#llm-inference#privacy 1 min read In your stack
  11. Defensive AI Research arXiv (AI security papers) Medium

    Tether: Verified Stateful Policy Engine for System-Level Monitoring of Agents

    Tether is a verified stateful policy engine that enforces security policies on the observed side-effects of AI agents executing shell commands and programs.

    ASI02 arXiv:2611.25599 #agents#policy-engine#runtime-enforcement 1 min read In your stack
  12. Attacks on AI Research arXiv (AI security papers) Medium

    ToolPulse: Side-Channel Eavesdropping on Tool-Using LLM Agents

    ToolPulse shows a side-channel attack can eavesdrop on tool-using LLM agents by exploiting local tool execution and shared system libraries on the user's machine.

    ReviewReview exposure of local tool execution on agent-connected user machines

    ASI02 arXiv:2611.25470 #side-channel#llm-agents#mcp 1 min read Brief In your stack

Reviewed 2026-11-09 · tools and news update automatically

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.