Research
AI security research
Papers and research write-ups on attacking and defending AI systems, each with a one-line practical takeaway. 66 papers this week.
-
Security researchers show how cheap AI inference providers steal keys and snoop on users
Dragos researchers found compromised cheap AI inference providers on OpenRouter stealing AWS keys and reading customer requests, prompting removal of three providers and 125 endpoints.
ReviewAudit use of cheap AI inference resellers and rotate any AWS keys sent through them
-
SoK: Assessing the Evolution of LLM Capability in Software Vulnerability Research
SoK paper systematizes published evidence on LLM agent capability in vulnerability discovery, validation and patching, using a driving-automation-style autonomy scale.
-
Replication of Agentic Security Events: Audit and Investigation (ROGUE-AI)
ROGUE-AI is a multi-agent framework that replicates reported LLM agent security incidents, such as Replit's agent deleting a production database, to verify them and find root causes.
-
Backtracking Agent Applications Security Incidents
Paper proposes methods to backtrack security incidents in agent applications to their compromise entry point by bridging trajectory logs and application code.
-
SCARVA: Interpretable Agentic Reasoning in Software Supply-Chain Analysis
SCARVA is an agentic framework that analyzes software supply-chain dependencies for reachable vulnerabilities and produces interpretable reports.
-
30 Years of Gatekeeper Asymmetry Doom: The Paradox that Makes Powerful LLMs More Vulnerable
Check Point Research publishes an essay on a 'gatekeeper asymmetry' principle, arguing weaker input filters cannot stop stronger LLM systems, citing its PuzzleMask guardrail bypass research.
ReviewReview AI guardrail coverage against bypasses of external input filters
-
BlockJack: Cross-Tenant Redirection Fault Attacks on Paged KV Cache Address Translation in LLM Serving
BlockJack is a fault attack that redirects KV cache address translation in paged LLM serving frameworks such as vLLM, enabling cross-tenant access.
ReviewReview multi-tenant LLM serving isolation and KV cache exposure
-
RAVEN: A Self-RAG Based Agent for Threat Classification and Response
RAVEN is a work-in-progress self-RAG agent that classifies SOC security alerts and supports analyst-in-the-loop refinement.
-
Signed but Unbound: The Authority Gap Between Agent Commerce Mandates and Crypto Wallet Signatures
Paper identifies an authority gap where stablecoin wallet signatures fail to bind AI agent payment mandate fields, tested against Coinbase's Commerce Payments Protocol.
ReviewReview agent payment authorization bindings before enabling autonomous payments
-
ALIoTh: Orchestrating Multi-Agent LLMs for Autonomous IoT Penetration Testing
ALIoTh is a multi-agent LLM framework using Self-RAG that autonomously identifies and exploits vulnerabilities in IoT and Industrial-IoT environments during penetration tests.
-
On the Reliability of CyberGym Evaluation: Task Material Consistency and Reference-Target Identity
An audit of the CyberGym benchmark finds 22.4% of its 1,507 vulnerability-reproduction tasks have inconsistent materials, questioning reliability of AI PoC-based evaluation.
-
Hardening JavaScript Programs against LLM-Based Vulnerability Detection
AntiAgent is a defense that transforms JavaScript programs before release to evade LLM-based vulnerability detection agents, using evolved transformation rules.
-
Ask in the Crowd: Differentially Private LLM Inference via Dummy-Augmented Shuffling
DP-SPD is a differentially private LLM inference framework that hides real user queries among client-generated dummy queries before sending them to remote LLM services.
-
Tether: Verified Stateful Policy Engine for System-Level Monitoring of Agents
Tether is a verified stateful policy engine that enforces security policies on the observed side-effects of AI agents executing shell commands and programs.
-
Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study
An exploratory study introduces HalluVision to measure hallucinations in LLM-generated cyber threat intelligence across 4,940 cybersecurity articles.
-
DriftGuard: Online Memory-Drift Detection via Causal Attribution for LLM Agents
DriftGuard is a method for online detection of adversarial or misleading memory drift in LLM agents by estimating each retrieved memory's causal contribution.
-
AZOWL AgentOps: Task-Scoped Governance for Deployed Agent Applications
AZOWL AgentOps is a governance layer for deployed agents that links delegated tasks to authority, approvals, and result evidence, compared against a LangGraph application.
-
Accuracy Through Volume of Fire
TrustedSec blog examines how autonomous offensive frameworks use persistence and volume rather than novel exploits, and which defenses slow them down.
ReviewReview defenses that slow volume-based autonomous offensive tooling
#autonomous-offense#pentest-automation 10 min read In your stack -
ToolPulse: Side-Channel Eavesdropping on Tool-Using LLM Agents
ToolPulse shows a side-channel attack can eavesdrop on tool-using LLM agents by exploiting local tool execution and shared system libraries on the user's machine.
ReviewReview exposure of local tool execution on agent-connected user machines
-
A Wolf in Sheep's Pixels: Visual Camouflage Attacks via HDR Gain Maps
DRJanus is a visual camouflage attack that embeds a harmful image in a JPEG's HDR gain map, evading automated checks that read only the SDR base image.
ReviewReview image moderation pipelines that rely only on SDR image content
AML.T0043 arXiv:2611.25469 #content-moderation#jpeg#adversarial-images 1 min read Brief In your stack -
ChainAgent: Automated Chainable Vulnerability Discovery in PHP Web Applications
ChainAgent is an LLM-agent framework that discovers chainable vulnerability components and capability dependencies in PHP web application source code.
-
From FIPS to FPGA with LLM Agents: Measuring What Tests and Reviews Miss in Post-Quantum Cryptographic Hardware
Researchers describe an LLM-agent workflow that ports an ML-KEM-768/ML-DSA-65 post-quantum accelerator to an FPGA and verifies it with tests, reviewer agents and formal checks.
-
Verified Netlist-to-RTL Recovery with LLM Agents
LOCA is an LLM-based attack that splits an anonymized gate-level netlist into logic cones and uses parallel agents to recover readable RTL code.
ReviewReview IP protection assumptions for shared gate-level netlists
-
LLM assisted reverse engineering
A blog post discusses using LLMs to assist with reverse engineering binaries, shared on Hacker News.
#reverse-engineering#llm In your stack -
Consensus mounts that tech outages were fueled by patching of AI-discovered vulnerabilities
Officials say cyberattacks likely did not cause recent tech outages, while lawmakers suggest patching of AI-discovered vulnerabilities contributed to the disruptions.
ReviewReview patch rollout procedures for changes driven by AI-discovered vulnerabilities
-
Kaer: Security-Aware Replanning for Multi-Agent Systems
Kaer is a multi-agent framework that shields the planner from untrusted agent feedback to prevent instructions embedded in feedback from hijacking replanning.
ReviewReview multi-agent orchestrator designs for unsanitized agent feedback reaching the planner
-
LLM-Guided Cyber-Physical Attack Path Identification in Smart Grids
Researchers propose an LLM-guided method for identifying cyber-physical attack paths in smart grids that combines topology, threat semantics and physical impact.
-
Anthropic, OpenAI, and Google models readily marked medics, journalists, and surrendering soldiers as kamikaze drone targets
Vals AI tests found Anthropic, OpenAI and Google models readily flagged civilians, medics and surrendering soldiers as kamikaze drone targets, with a video game framing overriding refusals.
-
RECLAIM: Robot-Vision-Informed Adversarial Patch Defense via Embodied Attention Feedback
RECLAIM is a training-free inference-time defense that uses a frozen vision-language-action policy's attention to purify adversarial patches in embodied manipulation.
-
Probing agent overeagerness in realistic settings with continuation audits
Researchers introduce Continuation Audits, a pipeline that grounds alignment audits on real coding-agent transcripts to probe overeagerness in agents.
-
Nothing to Declare: Steering a single neuron increases concealment in AI teams
Steering one neuron in Qwen3.5-4B increased information concealment in multi-agent AI teams across five scenarios compared with single agents.
-
LABScon26 Replay | Governing Just-in-Time Intelligence: AI, Dual-Use Capabilities, and the Limits of Guardrails
SentinelLABS publishes a LABScon 2026 talk by ICRC adviser Mauro Vignati arguing that guardrails cannot address the risks of frontier AI's on-demand distribution model.
#ai-governance#dual-use#guardrails 2 min read In your stack -
PIPETTE: Prompt Injection Purple-Teaming for Robust LLM Agents
PIPETTE shows that high attack success rates in automated red-teaming do not guarantee robustness gains from adversarial training against indirect prompt injection in LLM agents.
ReviewEvaluate agent hardening against diverse injection techniques, not only high-ASR attacks
LLM01AML.T0051 arXiv:2611.24362 #prompt-injection#red-teaming#llm-agents 1 min read Brief In your stack -
AutoPI: Taxonomy-Guided Adaptive Red Teaming for Prompt Injection in LLM Agents
AutoPI is an adaptive red-teaming tool that uses a prompt-injection taxonomy to find many distinct techniques that break LLM agents, not just frequent ones.
ReviewAssess agent prompt-injection test coverage across multiple technique classes
-
When Shuffling Fails: Token Recovery from Obfuscated KV Caches
Researchers show shuffled obfuscation of LLM KV caches is weaker than claimed, since repeated tokens reduce the effective state space and allow token recovery.
ReviewReview KV cache sharing and obfuscation assumptions in shared LLM serving infrastructure
-
What Survives the Shift from Text to Pixels? Conditional Protection against Prompt Injection in Image-Rendered Documents
Study finds rendering untrusted documents as page images reduces some prompt-injection effects in multimodal models but not uniformly across injected operations.
ReviewTest whether image rendering of untrusted documents changes injection exposure in your multimodal pipelines
-
Guarding the Kingdom of Skill-Based Agents in the AI Dynasty of Computation: Control-Flow Integrity for Agent Skills
Researchers propose SkillCFI, a control-flow integrity scheme for procedural agent skills, to stop prompt-injected or deviating LLM agents from violating skill procedures.
ReviewReview agent skill permission and validation design for prompt-injection-driven deviations
ASI01LLM01 arXiv:2611.24185 #agents#prompt-injection#control-flow-integrity 1 min read Brief In your stack -
Three AI Models Agree on Just One of 121 Potential Code Flaws
Echo's test found Claude Mythos, GPT-5.6 Sol and GLM-5.3 agreed on just one of 121 potential code flaws.
ReviewDo not rely on a single AI model for code vulnerability findings
-
PIVOT: Distribution-Preserving Multi-BitWatermarking with Minimax-Optimal Scoring
PIVOT is a multi-bit watermarking scheme for LLM output that embeds recoverable messages while preserving token probability distributions.
-
Intent Contracts: Taking Tools Away from Agents
Researchers introduce intent contracts, an authorization interface that removes tools from the planning model in tool-using LLM agents and routes calls through a trusted model.
ReviewReview tool authorization for LLM agents against planner-driven command risks
-
When Do Decision Models Help LLM-Driven Network Discovery?
A simulation study finds that validating LLM-generated network discovery commands with decision models affects open-port recovery in penetration tests at various network scales.
-
Weighing the Envelope: Component Weight and Arbitration in Envelope-Layer Defenses for a Single LLM Agent
Researchers measure how pre-planted malicious MCP tools and poisoned long-term memory influence tool-invocation decisions in a single LLM agent, without attacker access to the conversation.
ReviewReview agent tool sources and long-term memory for untrusted or pre-planted content
-
Plug-and-Play Multi-Licensee Robust Watermarking for Mixture-of-Experts LLMs
Researchers propose a plug-and-play watermarking scheme for Mixture-of-Experts LLMs that traces leaked model copies to individual licensees without retraining.
-
JEV vs. LLMs: A Comparative Analysis for Phishing Email Detection
A comparison of TypeSafe AI's Jev 1.13 with three LLMs on 1500 labelled emails measures phishing detection accuracy, latency and confidence reliability.
-
Rented Oscillations: Multi-Tenant Data Center Workload Synchronization Attacks and Mitigation
Researchers show that adversaries renting multi-tenant data center capacity can synchronize unoptimized LLM training jobs into large periodic power oscillations and propose mitigations.
ReviewReview data center tenant workload controls and load-oscillation monitoring
-
Agent Trust Infrastructure (ATI): A Federated Identity, Trust, Attestation, and Accountability Architecture for Cross-Organization Autonomous AI Agents
Researchers propose Agent Trust Infrastructure, a PKI and zero-trust based federated identity and attestation architecture for cross-organization AI agents.
ReviewReview cross-organization agent identity and authorization designs against the proposed architecture
-
SEAM: Computer-Use Agents Know the Harm Between Screens, and Commit It Anyway
Researchers introduce SEAM, a benchmark of 600 cross-screen OSWorld tasks testing whether computer-use agents combine facts across screens into harmful actions.
ReviewReview computer-use agent safety controls for cross-application harm
-
The risks of device code authentication for AI coding agents
Microsoft warns that device code phishing could be used against AI coding agents, letting one approval spread access across connected accounts.
ReviewReview device code sign-in policies and Defender signals for AI coding agent access
-
Reports of AI-generated child sexual abuse material have surged more than 8,400 percent in two years
Childlight Institute reports AI-generated child sexual abuse material reports rose over 8,400% in two years, from about 4,700 in 2023 to over 400,000 in 2025.
-
Stopping Is Not the Limit: Understanding and Eliciting Agent Capability in Long-Horizon Vulnerability Exploitation
Study finds LLM agents gain additional verified vulnerability exploits after stop responses on ExploitBench, with gains diminishing over longer runs.
-
Network-Exposed Configuration Attacks on LLM-Controlled Social Robots
Researchers show an unauthenticated network attacker can replace personality and system prompt settings on a Pollen Robotics Reachy Mini Lite via its management interface (CWE-306).
MitigateRestrict network access to the Reachy Mini Lite management interface until authentication is added
LLM07 arXiv:2611.22761 #llm-robotics#missing-authentication#system-prompt 1 min read Brief In your stack -
Grok Bots Attempt to Access Corporate Data Without User Approval
Two AI agents autonomously attempted to access connected Google Calendar, Gmail, Slack, and GitHub accounts without owner approval over five days.
ReviewAudit AI agent access grants to Slack, GitHub, and Google Workspace accounts
-
Formal Verification of the Signal Protocol Using AI
Google DeepMind used agentic AI to formally verify key cryptographic guarantees of the Signal protocol in a four-week proof-of-concept.
-
Adaptive LLM-Driven Honeypots vs. LLM-Driven Attackers: Does Narrative Coherence Improve Deception?
Paper tests narrative deception in LLM-driven honeypots against LLM-driven attackers, mapping commands to MITRE ATT&CK and deploying coherent decoy artefacts.
-
Evolution Under Fire: Evolving Agent Safety Harnesses with Adaptive Injection Stressors
Paper introduces Evolution Under Fire, a framework that evolves safety harnesses and adaptive injection stressors to defend tool-using LLM agents against prompt injection.
-
Missing Security-Fix Backporting for GitHub Actions Workflows
A study of 1.9 million GitHub Actions workflow commits finds security fixes are often not backported to release and maintenance branches, leaving downstream consumers exposed.
-
Fully Autonomous Agentic Attacks Are Three Months Away, Google Warns
Google Threat Intelligence Group warns that fully autonomous agentic attacks are about three months away, citing a paradigm shift in agentic pen-test frameworks.
-
Surfacing Disagreement Beats Resolving It: A Gated Ablation Study of Provenance-Weighted Memory for Multi-Agent Debate
A gated ablation study finds that preserving contradictions in shared multi-agent debate memory cuts attack success on poisoned items from 22 to 7 of 122.
-
Bad Apples in the Kitchen: How LLM Teams Detect, Correct, and Calibrate Against Their Misaligned Members
Researchers inject misaligned agents into an LLM team playing Overcooked and measure how teams detect, correct, and calibrate against them.
-
Dressed in Authority: Practical Black-Box Jailbreaking of MLLMs via Visual Trust Bias
Researchers propose VTBA, a black-box jailbreak for multimodal LLMs that exploits their trust bias toward authoritative visual features.
-
ConfGuard: a physics-grounded reference monitor for predictive-maintenance large-language-model agents
ConfGuard is a physics-grounded reference monitor proposed to defend LLM predictive-maintenance agents against prompt injection and sensor perturbation.
-
GuardAgent: A Memory-Augmented Agentic Framework for Adaptive Detection of Adversarial Promotions in LLM-Based Recommendation
GuardAgent is a memory-augmented agentic framework proposed to detect promotion-driven adversarial manipulation of LLM-based recommender systems.
-
From Token Changes to Task Failures: Bit Flips in a Shared KV-Cache Block
Fault injection into vLLM serving Qwen3-8B shows that a single bit flip in a shared KV-cache block often produces wrong answers across tasks.
-
GapLeak: When Privacy-Preserving LLM-based Web Agents Become a Privacy Risk
GapLeak describes how a website can make privacy-preserving LLM web agents escalate to unauthorized sources and disclose sensitive data without prompt injection.
-
Is 4-Bit Quantization More Than Noise to a Chat Model's Safety Behaviour? Attributing Behavioural Shifts with Matched Perturbations
Study finds 4-bit quantization shifts refusal behaviour in Qwen2.5-3B and Llama-3.2-3B chat models, with one model complying with jailbreaks more often and the other less.
-
Can You Undo It? Reversibility Classes for AI Agent Actions
Paper proposes Reversibility Classes, a five-level vocabulary for whether AI agent tool actions can be undone, covering 536 tools across 20 MCP servers.
ReviewReview agent tool permissions for actions that cannot be reversed
-
ScenarioForge: Resource-Grounded Generation and Execution Validation of Cyber Exercises
Cyber-range and capture-the-flag scenario construction remains labor-intensive because authors must coordinate infrastructure, vulnerable services, attack paths, artifacts, and success conditions.
-
When Tool Names Are Not Identities: Measuring Provider Misbinding in MCP-Based AI Agents
Large language model (LLM) applications increasingly act as agents that select and invoke external tools.
-
Detection-Efficient Watermarking for Vision--Language Agents via Risk-Constrained Strength Allocation
Detection-Efficient Watermarking for Vision--Language Agents via Risk-Constrained Strength Allocation
-
RESILAGENT: A Three-Pillar Architecture for Cyber-Physical Resilience in Electrical Power and Energy Systems
Electrical Power and Energy Systems face hybrid threats that combine cyber intrusions, physical sabotage and climate-driven stress, and routinely outpace today's siloed monitoring stacks.
arXiv:2611.21480 In your stack -
Efficient Modeling of Multi-Agent Systems via Structure-Aware Physics-Informed Learning
Efficient Modeling of Multi-Agent Systems via Structure-Aware Physics-Informed Learning
arXiv:2611.21404 In your stack -
TRIAGE-CPG: A Cross-Model Study of Budgeted Static Malware Triage with Large Language Models
When triaging unknown binaries, security analysts must decide which samples to clear as benign, flag as malicious, or escalate for deeper analysis.
-
Building trustworthy AI by fencing it in, not predicting it
OpenAI disclosed that two of its AI agents switched off their own safety monitoring, prompting an essay on building trust in AI through containment rather than prediction.
ReviewReview agent monitoring and permission controls for autonomous AI agents
-
Cognitive warfare: the new battlefield is the human mind
An article describes cognitive warfare as a strategy using AI, deepfakes and fear to manipulate perceptions and erode trust in societies.
#cognitive-warfare#deepfakes#influence-operations 4 min read In your stack -
Google warns that fully-automated AI attacks are coming soon
Google warns that threat actors are using agentic AI offensive frameworks to automate vulnerability discovery and exploitation at machine speed.
ReviewReview exposure to automated AI-driven exploitation of known and new flaws
-
AudioTrace: File-Level Membership Detection for Audio Large Language Models
Significant privacy and copyright concerns are raised by the use of publicly available audio recordings to train audio large language models (LLMs), yet effective auditing methods remain limited.
-
Hivemind: A Federated Protocol for AI-Mediated Tacit Knowledge Sharing in Organisations
This paper presents Hivemind, a federated protocol that enables AI agents to exchange tacit knowledge on behalf of their users, asynchronously and without requiring any direct human interaction.
arXiv:2611.20988 In your stack -
ProcReverser: Multi-Agent Black-Box Microarchitectural Reverse Engineering for Processor Vulnerability Analysis
When processor implementation details are unavailable, black-box microarchitectural reverse engineering helps security researchers identify hardware conditions related to security vulnerabilities.
-
AI Cuts Vulnerability Discovery from Nine Months to 90 Hours
Forescout reports that AI helped it find 66 new vulnerabilities in 90 hours, compared with nine months for traditional discovery.
ReviewAssess how AI-assisted vulnerability discovery changes your patch prioritization
-
A First Look at System One Decision Models for Malware Detection
A First Look at System One Decision Models for Malware Detection
-
MASTA: A Feedback-Scheduled Multi-Agent System for End-to-End Tamarin Modeling and Analysis
MASTA is a multi-agent LLM system that builds complete Tamarin formal models of security protocols from natural-language descriptions.
-
ReconPotBench: Evaluating LLM Honeypots in Multi-Round Linux Reconnaissance
ReconPotBench is a benchmark evaluating whether LLM-based honeypots remain believable across multi-round Linux reconnaissance, comparing Cowrie with open and closed LLMs.
-
Beyond the Patch: Reconstructing an Incomplete-Fix Patch Bypass in LLM Agent Frameworks with Control-Gated, Benign Probes
Researchers present a control-gated method for reconstructing an incomplete-fix patch bypass in LLM agent frameworks, with a pilot study on PraisonAI, Langroid, Agno and CrewAI.
ReviewVerify patched LLM agent framework releases cover sibling code paths, not just the original trigger