Learn / Detecting and responding to AI abuse
Lesson 6 of 69 minReviewed
Using AI in the SOC without inheriting its failure modes
Recognise how AI triage and investigation assistants fail (confabulation, automation bias, injection through alert data, excessive agency) and evaluate one on your own alerts before relying on it.
SOC teams use language models to summarise alerts, draft queries, enrich indicators and propose verdicts. The model brings the same properties to the SOC that it brings to any application: it states wrong things confidently, and it can be steered by text it reads. In a SOC, the text it reads is largely written by the attacker.
Failure modes and controls
| Failure mode | What it looks like | Control |
|---|---|---|
| Confabulation [1] | A confident verdict that cites a process, technique or log line that is not in the data | Every verdict must cite the queries and records it used; spot-check that they exist |
| Automation bias and overreliance [1, 2] | Analysts accept closures without checking because the assistant is usually right | Sample closed alerts for human review; track misses on seeded true positives |
| Injection through alert data [5, 6] | Email bodies, file names, command lines or user agents carry instructions the model follows | Treat alert fields as untrusted data; the assistant holds read-only access |
| Excessive agency [7] | The assistant can isolate hosts or disable accounts on its own judgement | Scoped credentials; containment actions need approval |
| Drift | Behaviour changes after a model or vendor update | Re-run the evaluation set after every update |
NIST's Generative AI Profile defines confabulation as confidently stated but false content, and describes automation bias, excessive deference to automated systems, as a risk that makes confabulation worse [1]. OWASP's entry on misinformation (LLM09) adds overreliance, trusting model output without verifying it, and recommends human oversight and automatic validation for high-stakes outputs [2]. Alert triage is high-stakes by definition: a wrong closure is a missed intrusion.
Injection is not theoretical here. GTIG reported FRUITSHELL, a PowerShell reverse shell containing hard-coded prompts intended to evade analysis by LLM-powered security tools [6]. The NCSC's advice for any system that processes untrusted input applies directly: assume the model can be confused, limit what it can do, and log its inputs, outputs and tool calls [5].
What public benchmarks say
CyberSOCEval, published in September 2025, benchmarks models on malware analysis and threat intelligence reasoning; its authors find current models far from saturating it, and report that test-time reasoning helps less there than in coding and mathematics [3]. The CTIBench authors note persistent concerns about reliability, accuracy and hallucination in threat intelligence tasks [4]. Benchmarks show the ceiling has not been reached. They do not tell you how a given product performs on your alerts, your log schema and your environment.
Evaluate before you trust
- Build a labelled sample of past alerts that includes known true positives, and record your analysts' verdicts and time-to-verdict on it.
- Run the assistant on the same sample. Make false negatives on the seeded true positives the primary metric; a closure rate alone rewards over-closing.
- Check that each verdict cites evidence an analyst can verify in a minute.
- Plant instructions in attacker-controlled fields (a mail subject, a file name) and see whether the assistant follows them.
- Keep response actions behind approval with separately scoped credentials, and repeat the evaluation after each model or vendor change.
Try it on ThreatDeck
- Evaluate an AI SOC or agentic triage tool The evaluation checklist from this lesson, with progress saved in your browser.
- Misinformation (LLM09) Read the OWASP entry and recent stories on overreliance and hallucination.
- Inspect (radar) An open-source framework for building your own evaluations on labelled alert samples.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
Closure rate and speed reward over-closing. The failure that matters is a missed intrusion, which only shows up when you measure misses on known true positives.
-
Much of the data a SOC analyses is written by the attacker. GTIG has reported malware with embedded prompts aimed at LLM-based analysis, so these fields are prompt injection inputs.
-
Automation bias is excessive deference to an automated system. It turns occasional confabulation into missed incidents, which is why closed alerts need sampled human review.
-
Read-only access limits what an injected or confabulating assistant can break. Containment can be added later behind approval once evaluation shows the verdicts hold up.
References
- NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
- LLM09:2025 Misinformation. OWASP GenAI Security Project, 2024-11.
- CyberSOCEval: Benchmarking LLMs Capabilities for Malware Analysis and Threat Intelligence Reasoning. Deason et al. (arXiv:2509.20166), 2025-09.
- CTIBench: A Benchmark for Evaluating LLMs in Cyber Threat Intelligence. Alam et al. (arXiv:2406.07599), 2024-06.
- Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.
- GTIG AI Threat Tracker: Advances in Threat Actor Usage of AI Tools. Google Threat Intelligence Group, 2025-11-06.
- LLM06:2025 Excessive Agency. OWASP GenAI Security Project, 2024-11.