threatdeck
Live 13:45 UTC ·

Learn / Detecting and responding to AI abuse

Lesson 6 of 69 minReviewed

Using AI in the SOC without inheriting its failure modes

Recognise how AI triage and investigation assistants fail (confabulation, automation bias, injection through alert data, excessive agency) and evaluate one on your own alerts before relying on it.

SOC teams use language models to summarise alerts, draft queries, enrich indicators and propose verdicts. The model brings the same properties to the SOC that it brings to any application: it states wrong things confidently, and it can be steered by text it reads. In a SOC, the text it reads is largely written by the attacker.

Failure modes and controls

How AI assistants fail in security operations
Failure modeWhat it looks likeControl
Confabulation [1]A confident verdict that cites a process, technique or log line that is not in the dataEvery verdict must cite the queries and records it used; spot-check that they exist
Automation bias and overreliance [1, 2]Analysts accept closures without checking because the assistant is usually rightSample closed alerts for human review; track misses on seeded true positives
Injection through alert data [5, 6]Email bodies, file names, command lines or user agents carry instructions the model followsTreat alert fields as untrusted data; the assistant holds read-only access
Excessive agency [7]The assistant can isolate hosts or disable accounts on its own judgementScoped credentials; containment actions need approval
DriftBehaviour changes after a model or vendor updateRe-run the evaluation set after every update

NIST's Generative AI Profile defines confabulation as confidently stated but false content, and describes automation bias, excessive deference to automated systems, as a risk that makes confabulation worse [1]. OWASP's entry on misinformation (LLM09) adds overreliance, trusting model output without verifying it, and recommends human oversight and automatic validation for high-stakes outputs [2]. Alert triage is high-stakes by definition: a wrong closure is a missed intrusion.

Injection is not theoretical here. GTIG reported FRUITSHELL, a PowerShell reverse shell containing hard-coded prompts intended to evade analysis by LLM-powered security tools [6]. The NCSC's advice for any system that processes untrusted input applies directly: assume the model can be confused, limit what it can do, and log its inputs, outputs and tool calls [5].

What public benchmarks say

CyberSOCEval, published in September 2025, benchmarks models on malware analysis and threat intelligence reasoning; its authors find current models far from saturating it, and report that test-time reasoning helps less there than in coding and mathematics [3]. The CTIBench authors note persistent concerns about reliability, accuracy and hallucination in threat intelligence tasks [4]. Benchmarks show the ceiling has not been reached. They do not tell you how a given product performs on your alerts, your log schema and your environment.

Evaluate before you trust

  1. Build a labelled sample of past alerts that includes known true positives, and record your analysts' verdicts and time-to-verdict on it.
  2. Run the assistant on the same sample. Make false negatives on the seeded true positives the primary metric; a closure rate alone rewards over-closing.
  3. Check that each verdict cites evidence an analyst can verify in a minute.
  4. Plant instructions in attacker-controlled fields (a mail subject, a file name) and see whether the assistant follows them.
  5. Keep response actions behind approval with separately scoped credentials, and repeat the evaluation after each model or vendor change.

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. Which metric best shows whether an AI triage assistant is safe to rely on?
  2. Why are alert fields such as email subjects, file names and user agents a security concern for an AI analyst?
  3. NIST AI 600-1 describes automation bias. In a SOC, what does it look like?
  4. What should an AI investigation assistant be allowed to do on day one?

References

  1. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
  2. LLM09:2025 Misinformation. OWASP GenAI Security Project, 2024-11.
  3. CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning. Deason et al. (arXiv:2509.20166), 2025-09.
  4. CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence. Alam et al. (arXiv:2406.07599), 2024-06.
  5. Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.
  6. GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools. Google Threat Intelligence Group, 2025-11-06.
  7. LLM06:2025 Excessive Agency. OWASP GenAI Security Project, 2024-11.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.