threatdeck
Live 13:45 UTC ·

Learn / Detecting and responding to AI abuse

Lesson 3 of 69 minReviewed

Detection engineering for AI abuse

Turn reported attacker behaviour into rules on logs you already collect, choose between alert, baseline and behavioural rules, and write false positives down instead of hiding them.

Most AI abuse is visible in logs a SOC already collects: cloud audit trails, EDR process, file and network events, and proxy logs. The work is in encoding one observed behaviour per rule and keeping the source that reported it next to the query, so the next analyst knows why the rule exists. The detections in ThreatDeck's library follow that pattern; a few examples:

From reported behaviour to rule
BehaviourSignalExample rule
Stolen model key being tested [1]Failed runtime calls across several models or regionsInvocation probing
Attacker enabling models with a stolen IAM user key [2]Console-style model-access APIs called with a long-term keyModel access from access key
Malware driving an installed AI CLI to find secrets [3]AI CLI with an approval-bypass flag, non-interactive, launched by a script runtimeScripted AI CLI with bypass
Malware asking a model API to rewrite its own code [4]wscript, cscript or mshta connecting to a model APIScript host calls LLM API
Rogue or poisoned local MCP server [5]MCP config written by another process; rare egress from a server processMCP config modified

Three ways to turn a behaviour into a rule

  • Rare by nature: alert. Some events almost never happen legitimately. Deleting Bedrock invocation logging, which LLMjacking operators do before abuse [1], and a Windows script host talking to a model API both fit. Low volume, high signal, safe to page on.
  • Rare for this principal: baseline. Many abuse signals are normal somewhere in the company. A first inference call by a principal in 14 days, invocation volume far above that principal's own 13-day average, or a new caller listing Azure OpenAI keys are only interesting relative to history. Exclude the window you are scoring from the baseline, or a spike raises its own threshold.
  • Shape, not indicator: behavioural. Anthropic published no host indicators for the AI-orchestrated espionage campaign it disrupted, only that the agent issued thousands of requests, often several per second [6]. A rule such as offensive tooling burst therefore looks for machine-paced runs of scanners and credential tools under an agent process, not for a hash.

False positives belong in the rule

AI tools legitimately do much of what attackers do with them, so every rule needs its false positives written next to it. Developers run coding agents with approval bypass enabled, which is why a rule that matches the bypass flag alone is an inventory hunt, while the s1ngularity rule also requires non-interactive mode and a script-runtime parent: the malicious Nx versions invoked claude, gemini and q with --dangerously-skip-permissions, --yolo and --trust-all-tools from a postinstall script [3]. Corporate NAT and shared CI runners make many keys share one source address. A git clone writes .mcp.json. Express exclusions as a named, reviewed allowlist (gateway egress ranges, CI runners, the MDM tool that deploys approved MCP configs) rather than an unexplained filter.

Keep a rule at hunt maturity until you have measured its false-positive rate on your own data, then promote it. Re-check after AI tool releases: flag names, config file locations and telemetry attributes change between versions, and the MCP project notes that client configuration files live in known per-platform locations precisely so endpoint tooling can inventory them [5].

Behaviour-based rules age better than indicators. Google's threat intelligence group describes malware that asks a model to regenerate its own code, which defeats static signatures by design [4]; the network connection from a script host to a model API stays the same across variants.

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. Which signal is the best candidate for an alert that pages someone, rather than a hunt?
  2. Why does the s1ngularity rule require a script-runtime parent and non-interactive mode in addition to the bypass flag?
  3. A per-principal volume baseline is computed over the last 14 days, including the hour being scored. What goes wrong?
  4. Your Vertex AI new-principal hunt has been silent for months. What should you check first?

References

  1. LLMjacking: Stolen Cloud Credentials Used in New AI Attack. Sysdig Threat Research Team, 2024-05-06.
  2. Hijacking AI infrastructure with non-human identities like access tokens. Permiso Security, 2024-10-03.
  3. s1ngularity: supply chain attack leaks secrets on GitHub: everything you need to know. Wiz Research, 2025-08.
  4. GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools. Google Threat Intelligence Group, 2025-11-06.
  5. MCP Local Server Security. Model Context Protocol, 2026-07-28.
  6. Disrupting an AI-orchestrated cyber espionage campaign. Anthropic, 2025-11-13.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.