threatdeck
Live 13:45 UTC ·

Learn / Detecting and responding to AI abuse

Lesson 5 of 69 minReviewed

AI-enabled attackers: what has been reported and what changes

Separate documented attacker use of AI from speculation, and adjust detection and response for machine-paced intrusions and malware that calls model APIs.

Claims about AI-powered attacks are easy to find; primary reports with detail are fewer. The cases below come from the model providers and threat intelligence teams that observed them, and each includes the limits its authors stated.

What has been documented

  • Agent-run intrusions. In August 2025 Anthropic described a data-extortion operation that used Claude Code to automate reconnaissance, credential harvesting and network penetration against at least 17 organisations, including healthcare and emergency services; the model also helped choose what to steal and drafted ransom demands, some above $500,000 [3].
  • AI-orchestrated espionage. In November 2025 Anthropic reported a campaign, detected in mid-September and attributed with high confidence to a Chinese state-sponsored group, against about 30 targets. AI performed an estimated 80–90% of the work, with humans at perhaps four to six decision points; at peak it issued thousands of requests, often several per second. The operators bypassed safeguards by splitting the work into small tasks and claiming to be a security firm doing defensive testing [1].
  • Malware that calls a model at run time. Google's threat intelligence group (GTIG) reported PROMPTSTEAL, attributed to APT28, which queries an open-weight coding model through the Hugging Face API to generate Windows commands that collect documents, and PROMPTFLUX, a VBScript dropper that asks the Gemini API for new obfuscated code and saves the result to the Startup folder [2]. GTIG also describes QUIETVAULT, a credential stealer that uses on-host AI CLI tools to look for more secrets [2].

The same reports set limits. Anthropic notes that the model sometimes hallucinated credentials or claimed to have found secrets that were in fact public, which it calls an obstacle to fully autonomous attacks [1]. GTIG assesses PROMPTFLUX as experimental, with self-update code commented out and no ability yet to compromise a network [2]. Plan for faster and cheaper versions of known techniques, not for a new class of attack.

What changes for defenders

Reported behaviour, defender response and ATLAS mapping [4]
BehaviourWhat to doATLAS
Machine-paced reconnaissance and exploitationDetect on rate and shape (many tools, many hosts, short intervals) instead of single indicators; automate the first containment stepAML.T0116, AML.T0124
Malware generating commands through a modelRestrict which processes may reach model APIs; alert on script hosts and unsigned binaries that doAML.T0102, AML.T0096
Code regenerated on each runPrefer behavioural detection to hashes and static signatures [2]AML.T0102
Stolen API keys used by malwareRotate exposed keys; route AI traffic through a gateway so direct calls stand out [2]AML.T0096

Speed matters most. A campaign that issues several requests per second [1] compresses the time between initial access and impact, so a containment action that waits for a morning review is too slow. Pre-approve narrow automatic actions, such as isolating a host or disabling a session token, for high-confidence detections, and keep broader actions behind a human.

The hallucination limit [1] is also a signal. Failed logins with credentials that never existed, or bursts of requests for resources that do not exist, look different from a human operator who already knows the environment.

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. In the espionage campaign Anthropic reported in November 2025, how did the operators get the model to cooperate?
  2. Which statement about PROMPTFLUX matches GTIG's assessment?
  3. What is the most useful detection change for machine-paced intrusions?
  4. Why can hallucinated output from an attacker's agent help defenders?

References

  1. Disrupting an AI-orchestrated cyber espionage campaign. Anthropic, 2025-11-13.
  2. GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools. Google Threat Intelligence Group, 2025-11-06.
  3. Detecting and countering misuse of AI: August 2025. Anthropic, 2025-08-27.
  4. MITRE ATLAS (Adversarial Threat Landscape for AI Systems). MITRE.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.