Learn / Detecting and responding to AI abuse
Lesson 5 of 69 minReviewed
AI-enabled attackers: what has been reported and what changes
Separate documented attacker use of AI from speculation, and adjust detection and response for machine-paced intrusions and malware that calls model APIs.
Claims about AI-powered attacks are easy to find; primary reports with detail are fewer. The cases below come from the model providers and threat intelligence teams that observed them, and each includes the limits its authors stated.
What has been documented
- Agent-run intrusions. In August 2025 Anthropic described a data-extortion operation that used Claude Code to automate reconnaissance, credential harvesting and network penetration against at least 17 organisations, including healthcare and emergency services; the model also helped choose what to steal and drafted ransom demands, some above $500,000 [3].
- AI-orchestrated espionage. In November 2025 Anthropic reported a campaign, detected in mid-September and attributed with high confidence to a Chinese state-sponsored group, against about 30 targets. AI performed an estimated 80–90% of the work, with humans at perhaps four to six decision points; at peak it issued thousands of requests, often several per second. The operators bypassed safeguards by splitting the work into small tasks and claiming to be a security firm doing defensive testing [1].
- Malware that calls a model at run time. Google's threat intelligence group (GTIG) reported PROMPTSTEAL, attributed to APT28, which queries an open-weight coding model through the Hugging Face API to generate Windows commands that collect documents, and PROMPTFLUX, a VBScript dropper that asks the Gemini API for new obfuscated code and saves the result to the Startup folder [2]. GTIG also describes QUIETVAULT, a credential stealer that uses on-host AI CLI tools to look for more secrets [2].
The same reports set limits. Anthropic notes that the model sometimes hallucinated credentials or claimed to have found secrets that were in fact public, which it calls an obstacle to fully autonomous attacks [1]. GTIG assesses PROMPTFLUX as experimental, with self-update code commented out and no ability yet to compromise a network [2]. Plan for faster and cheaper versions of known techniques, not for a new class of attack.
What changes for defenders
| Behaviour | What to do | ATLAS |
|---|---|---|
| Machine-paced reconnaissance and exploitation | Detect on rate and shape (many tools, many hosts, short intervals) instead of single indicators; automate the first containment step | AML.T0116, AML.T0124 |
| Malware generating commands through a model | Restrict which processes may reach model APIs; alert on script hosts and unsigned binaries that do | AML.T0102, AML.T0096 |
| Code regenerated on each run | Prefer behavioural detection to hashes and static signatures [2] | AML.T0102 |
| Stolen API keys used by malware | Rotate exposed keys; route AI traffic through a gateway so direct calls stand out [2] | AML.T0096 |
Speed matters most. A campaign that issues several requests per second [1] compresses the time between initial access and impact, so a containment action that waits for a morning review is too slow. Pre-approve narrow automatic actions, such as isolating a host or disabling a session token, for high-confidence detections, and keep broader actions behind a human.
The hallucination limit [1] is also a signal. Failed logins with credentials that never existed, or bursts of requests for resources that do not exist, look different from a human operator who already knows the environment.
Try it on ThreatDeck
- Prepare for AI-powered attackers Checklist for detection speed, egress control and response automation.
- Script host calls an LLM API The PROMPTFLUX and PROMPTSTEAL pattern on Windows endpoints.
- Offensive tooling burst under an AI agent A shape-based hunt for machine-paced activity.
- AI API requests bypassing the gateway Find model API traffic that does not come from your sanctioned path.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
Anthropic describes task decomposition plus a false defensive-testing pretext. Each request lacked the full malicious context, which is why provider-side safeguards are not a substitute for your own detection.
-
GTIG calls PROMPTFLUX experimental, with self-update code commented out. PROMPTSTEAL is the family attributed to APT28 and was observed in operations.
-
Reports describe thousands of requests, often several per second, and no published host indicators. Rate-and-shape rules plus fast, narrow containment address the speed; hashes and provider refusals do not.
-
Anthropic reports that the model invented credentials and overstated findings. Attempts against non-existent accounts and resources stand out against normal operator behaviour and can be detected.
References
- Disrupting an AI-orchestrated cyber espionage campaign. Anthropic, 2025-11-13.
- GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools. Google Threat Intelligence Group, 2025-11-06.
- Detecting and countering misuse of AI: August 2025. Anthropic, 2025-08-27.
- MITRE ATLAS (Adversarial Threat Landscape for AI Systems). MITRE.