threatdeck
Live 13:45 UTC ·

All playbooks

PlaybookSOC leads, detection engineering, CISOs

Evaluate an AI SOC or agentic triage tool

Vendors pitch AI analysts that triage alerts and run investigations. Practitioners are sceptical: bolting a model onto SOAR is not an agentic SOC, and studies of LLMs classifying network traffic report high false-positive rates on benign data. The question is whether a tool reduces analyst time on your alerts without hiding misses.

What practitioners converge on

  1. Test on your own historical alerts. Replay a labelled sample of past alerts, including known true positives, and compare verdicts and time-to-verdict with your analysts.
  2. Measure misses, not just closes. Track false negatives on seeded true positives as the primary metric; closure rate alone rewards over-closing.
  3. Require evidence in every verdict. Each decision should cite the queries, logs and enrichment it used so an analyst can check it in a minute.
  4. Scope the agent's actions. Start read-only; any containment action (isolate host, disable user) needs approval and its own scoped credentials.
  5. Compare vendors on the same sample. Run commercial assistants (for example Microsoft Security Copilot, or the AI SOC vendors listed below) against the same labelled sample and score them with the same metrics as your own evals.

Checklist

7 steps
  • LLM09
  • LLM01
  • LLM06
  • ASI08

Saved in this browser. Track technique-level testing per system in Coverage.

Open-source stack

  • Inspect AdoptBuild your own evals on labelled alert samples

If you are evaluating vendors

AI SOC / agentic SOC

  • On our historical alerts with known outcomes, what share of true and false positives did it classify correctly?
  • Does every verdict show the evidence and queries it ran, so an analyst can verify it?
  • Which actions can it take on its own (containment, account disable), and what approval gates exist?
  • What read and write permissions does it need in our tools, and how are those credentials stored?

Full vendor landscape and POC questions

Detect and hunt 4

Techniques

References

Latest on this

  1. Defensive AI Research arXiv (AI security papers)

    Measuring Hallucination in Large Language Models for Cyber Threat Intelligence: An Exploratory Study

    An exploratory study introduces HalluVision to measure hallucinations in LLM-generated cyber threat intelligence across 4,940 cybersecurity articles.

    LLM09 arXiv:2611.25590 #cti#hallucination#llm In your stack
  2. Defensive AI Research arXiv (AI security papers) Medium

    Intent Contracts: Taking Tools Away from Agents

    Researchers introduce intent contracts, an authorization interface that removes tools from the planning model in tool-using LLM agents and routes calls through a trusted model.

    ReviewReview tool authorization for LLM agents against planner-driven command risks

    ASI02LLM06 arXiv:2611.24025 #agents#authorization#tool-use 1 min read In your stack

Reviewed 2026-11-09 · tools and news update automatically

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.