PlaybookSOC leads, detection engineering, CISOs
Evaluate an AI SOC or agentic triage tool
Vendors pitch AI analysts that triage alerts and run investigations. Practitioners are sceptical: bolting a model onto SOAR is not an agentic SOC, and studies of LLMs classifying network traffic report high false-positive rates on benign data. The question is whether a tool reduces analyst time on your alerts without hiding misses.
What practitioners converge on
- Test on your own historical alerts. Replay a labelled sample of past alerts, including known true positives, and compare verdicts and time-to-verdict with your analysts.
- Measure misses, not just closes. Track false negatives on seeded true positives as the primary metric; closure rate alone rewards over-closing.
- Require evidence in every verdict. Each decision should cite the queries, logs and enrichment it used so an analyst can check it in a minute.
- Scope the agent's actions. Start read-only; any containment action (isolate host, disable user) needs approval and its own scoped credentials.
- Compare vendors on the same sample. Run commercial assistants (for example Microsoft Security Copilot, or the AI SOC vendors listed below) against the same labelled sample and score them with the same metrics as your own evals.
Checklist
7 stepsSaved in this browser. Track technique-level testing per system in Coverage.
Open-source stack
- Inspect AdoptBuild your own evals on labelled alert samples
If you are evaluating vendors
AI SOC / agentic SOC
- On our historical alerts with known outcomes, what share of true and false positives did it classify correctly?
- Does every verdict show the evidence and queries it ran, so an analyst can verify it?
- Which actions can it take on its own (containment, account disable), and what approval gates exist?
- What read and write permissions does it need in our tools, and how are those credentials stored?
Detect and hunt 4
Techniques
References
Latest on this
-
Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study
An exploratory study introduces HalluVision to measure hallucinations in LLM-generated cyber threat intelligence across 4,940 cybersecurity articles.
-
Intent Contracts: Taking Tools Away from Agents
Researchers introduce intent contracts, an authorization interface that removes tools from the planning model in tool-using LLM agents and routes calls through a trusted model.
ReviewReview tool authorization for LLM agents against planner-driven command risks
Reviewed 2026-11-09 · tools and news update automatically