Learn / Detecting and responding to AI abuse
Lesson 3 of 69 minReviewed
Detection engineering for AI abuse
Turn reported attacker behaviour into rules on logs you already collect, choose between alert, baseline and behavioural rules, and write false positives down instead of hiding them.
Most AI abuse is visible in logs a SOC already collects: cloud audit trails, EDR process, file and network events, and proxy logs. The work is in encoding one observed behaviour per rule and keeping the source that reported it next to the query, so the next analyst knows why the rule exists. The detections in ThreatDeck's library follow that pattern; a few examples:
| Behaviour | Signal | Example rule |
|---|---|---|
| Stolen model key being tested [1] | Failed runtime calls across several models or regions | Invocation probing |
| Attacker enabling models with a stolen IAM user key [2] | Console-style model-access APIs called with a long-term key | Model access from access key |
| Malware driving an installed AI CLI to find secrets [3] | AI CLI with an approval-bypass flag, non-interactive, launched by a script runtime | Scripted AI CLI with bypass |
| Malware asking a model API to rewrite its own code [4] | wscript, cscript or mshta connecting to a model API | Script host calls LLM API |
| Rogue or poisoned local MCP server [5] | MCP config written by another process; rare egress from a server process | MCP config modified |
Three ways to turn a behaviour into a rule
- Rare by nature: alert. Some events almost never happen legitimately. Deleting Bedrock invocation logging, which LLMjacking operators do before abuse [1], and a Windows script host talking to a model API both fit. Low volume, high signal, safe to page on.
- Rare for this principal: baseline. Many abuse signals are normal somewhere in the company. A first inference call by a principal in 14 days, invocation volume far above that principal's own 13-day average, or a new caller listing Azure OpenAI keys are only interesting relative to history. Exclude the window you are scoring from the baseline, or a spike raises its own threshold.
- Shape, not indicator: behavioural. Anthropic published no host indicators for the AI-orchestrated espionage campaign it disrupted, only that the agent issued thousands of requests, often several per second [6]. A rule such as offensive tooling burst therefore looks for machine-paced runs of scanners and credential tools under an agent process, not for a hash.
False positives belong in the rule
AI tools legitimately do much of what attackers do with them, so every rule needs its false positives written next to it. Developers run coding agents with approval bypass enabled, which is why a rule that matches the bypass flag alone is an inventory hunt, while the s1ngularity rule also requires non-interactive mode and a script-runtime parent: the malicious Nx versions invoked claude, gemini and q with --dangerously-skip-permissions, --yolo and --trust-all-tools from a postinstall script [3]. Corporate NAT and shared CI runners make many keys share one source address. A git clone writes .mcp.json. Express exclusions as a named, reviewed allowlist (gateway egress ranges, CI runners, the MDM tool that deploys approved MCP configs) rather than an unexplained filter.
Keep a rule at hunt maturity until you have measured its false-positive rate on your own data, then promote it. Re-check after AI tool releases: flag names, config file locations and telemetry attributes change between versions, and the MCP project notes that client configuration files live in known per-platform locations precisely so endpoint tooling can inventory them [5].
Behaviour-based rules age better than indicators. Google's threat intelligence group describes malware that asks a model to regenerate its own code, which defeats static signatures by design [4]; the network connection from a script host to a model API stays the same across variants.
Try it on ThreatDeck
- Detection & hunting library Filter by log source and maturity; each rule lists its sources and false positives.
- AI CLI started with an approval-bypass flag Compare this inventory hunt with the stricter s1ngularity rule.
- One source IP using several AWS principals Read how the rule handles NAT and VPN egress as false positives.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
Deleting the logging configuration is rare outside planned changes and is a documented LLMjacking step. The other signals occur routinely in legitimate work and need a baseline or more context before anyone is paged.
-
The flag alone is an inventory of risky habits. The malicious Nx postinstall script launched AI CLIs non-interactively from a script runtime, and that combination is what separates the attack from a developer's own session.
-
If the anomalous window is part of the baseline, it inflates the average and shrinks the deviation. Score the recent window against history that excludes it.
-
The rule depends on an opt-in log. A companion check that the log is flowing is what tells you silence means no activity rather than no data.
References
- LLMjacking: Stolen Cloud Credentials Used in New AI Attack. Sysdig Threat Research Team, 2024-05-06.
- Hijacking AI infrastructure with non-human identities like access tokens. Permiso Security, 2024-10-03.
- s1ngularity: supply chain attack leaks secrets on GitHub: everything you need to know. Wiz Research, 2025-08.
- GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools. Google Threat Intelligence Group, 2025-11-06.
- MCP Local Server Security. Model Context Protocol, 2026-07-28.
- Disrupting an AI-orchestrated cyber espionage campaign. Anthropic, 2025-11-13.