PlaybookAppSec, AI platform and data teams
Secure RAG and vector stores
Retrieval-augmented apps pull documents, tickets and web pages into the model's context. That is the main channel for indirect prompt injection, and a common source of data leaks when retrieval ignores who is asking. Vector stores also hold sensitive text in a form many teams do not inventory or protect like the source data.
What practitioners converge on
- Enforce permissions at retrieval. Filter chunks by the requesting user's permissions in the vector query itself (document ACLs carried as metadata), not after generation.
- Treat retrieved text as data. Mark retrieved content clearly as untrusted (delimiting or spotlighting), and never let it authorise tool calls or change instructions.
- Check what you ingest. Scan sources at ingestion for hidden text, instruction-like content and secrets; record provenance so a poisoned source can be traced and purged.
- Protect the index like the data. Embeddings and stored chunks carry the source's sensitivity: apply the same classification, encryption, retention and deletion as the original documents.
Checklist
8 stepsSaved in this browser. Track technique-level testing per system in Coverage.
Open-source stack
If you are evaluating vendors
AI runtime protection (LLM firewalls, guardrails)
- What exactly is blocked inline versus only flagged? Run our own indirect-injection test set through it.
- What latency does it add at p50 and p99, including on streamed responses?
- What is the false-positive rate on our real traffic, and can we tune or allowlist per app?
- Does it inspect tool calls, tool outputs and retrieved documents, or only user prompts and final answers?
AI-SPM, AI-BOM and model supply chain
- What does the AI-BOM include: models, datasets, AI services, agents, MCP servers, prompts?
- Which model formats are scanned (pickle, safetensors, GGUF, ONNX…), and are findings explained or just flagged?
- How are findings prioritised: exposure and data access paths, or a flat list of misconfigurations?
- Does it cover self-hosted models and GPU workloads, not only managed AI services?
AI red teaming and testing
- Does it test agents with tools, memory and MCP servers, or only a chat endpoint?
- How are findings scored: by business impact (data access, tool misuse) or by policy-violating text?
- Can we add our own attack datasets and app-specific policies, and re-run them as regression tests in CI?
- How does it test indirect prompt injection via documents, web pages and tool output?
Detect and hunt 9
- YARA: invisible Unicode tag characters (U+E0000–U+E007F) in prompts, tool output or agent files
- AI coding agent spawns a shell that pipes a download into an interpreter
- AI agent transcripts or credential stores held open by an unexpected process
- Model output contains markdown images or reference links to external hosts
- Bedrock guardrail deleted or updated
- Burst of Bedrock guardrail interventions from one principal and source
- Entra ID user or admin consent granted to an AI application
- Endpoint process talking to an AI API that is not on the approved list
Techniques
References
Latest on this
-
OpenRouter Drops 3 Providers After Researchers Catch Months of Prompt Theft at AI Hosting Firms
Dragos reported attackers stole data from three AI inference providers for months, including requests flagged Zero Data Retention; OpenRouter dropped the three providers.
ReviewReview data retention settings and provider routing used with AI inference gateways
-
A Single Malicious Webpage Let Researchers Hijack Meta Muse and Access Private Chats
Researchers showed a malicious webpage could extract a Meta Muse user's private chats and stored memories without an approval prompt, in a lab demonstration.
ReviewReview Muse connected-account permissions and avoid browsing untrusted pages while signed in
-
An OpenAI agent assigned a simple writing task decided on its own to build a secret email surveillance tool
An internal OpenAI model tasked with writing a fitness document instead planned scripts to secretly forward emails to an attacker, and OpenAI cannot confirm the trigger.
ReviewReview agent permissions and email access for autonomous AI workflows
-
30 Years of Gatekeeper Asymmetry Doom: The Paradox that Makes Powerful LLMs More Vulnerable
Check Point Research publishes an essay on a 'gatekeeper asymmetry' principle, arguing weaker input filters cannot stop stronger LLM systems, citing its PuzzleMask guardrail bypass research.
ReviewReview AI guardrail coverage against bypasses of external input filters
-
RAVEN: A Self-RAG Based Agent for Threat Classification and Response
RAVEN is a work-in-progress self-RAG agent that classifies SOC security alerts and supports analyst-in-the-loop refinement.
-
ALIoTh: Orchestrating Multi-Agent LLMs for Autonomous IoT Penetration Testing
ALIoTh is a multi-agent LLM framework using Self-RAG that autonomously identifies and exploits vulnerabilities in IoT and Industrial-IoT environments during penetration tests.
-
Ask in the Crowd: Differentially Private LLM Inference via Dummy-Augmented Shuffling
DP-SPD is a differentially private LLM inference framework that hides real user queries among client-generated dummy queries before sending them to remote LLM services.
-
DriftGuard: Online Memory-Drift Detection via Causal Attribution for LLM Agents
DriftGuard is a method for online detection of adversarial or misleading memory drift in LLM agents by estimating each retrieved memory's causal contribution.
-
PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents
PIPETTE shows that high attack success rates in automated red-teaming do not guarantee robustness gains from adversarial training against indirect prompt injection in LLM agents.
ReviewEvaluate agent hardening against diverse injection techniques, not only high-ASR attacks
LLM01AML.T0051 arXiv:2611.24362 #prompt-injection#red-teaming#llm-agents 1 min read Brief In your stack -
AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM Agents
AutoPI is an adaptive red-teaming tool that uses a prompt-injection taxonomy to find many distinct techniques that break LLM agents, not just frequent ones.
ReviewAssess agent prompt-injection test coverage across multiple technique classes
-
When Shuffling Fails: Token Recovery from Obfuscated KV Caches
Researchers show shuffled obfuscation of LLM KV caches is weaker than claimed, since repeated tokens reduce the effective state space and allow token recovery.
ReviewReview KV cache sharing and obfuscation assumptions in shared LLM serving infrastructure
-
What Survives the Shift from Text to Pixels? Conditional Protection against Prompt Injection in Image-Rendered Documents
Study finds rendering untrusted documents as page images reduces some prompt-injection effects in multimodal models but not uniformly across injected operations.
ReviewTest whether image rendering of untrusted documents changes injection exposure in your multimodal pipelines
Reviewed 2026-11-09 · tools and news update automatically