Learn / Red teaming AI systems
Lesson 4 of 68 minReviewed
Choosing red team tooling: garak, PyRIT, promptfoo, Inspect
Match open-source tools to the job (scanning an endpoint, orchestrating multi-turn campaigns, testing an application in CI, building custom evaluations) and know the limits of automated results.
Open-source tooling for AI red teaming has settled into a few shapes. None of them is a substitute for scoping and threat modelling, and all of them produce results that need a person to interpret. Microsoft's AI red team puts it directly: adopt tools such as PyRIT to scale up, but keep humans in the red teaming loop [5].
| Tool | Shape | Building blocks | Typical output |
|---|---|---|---|
| garak (NVIDIA) | Vulnerability scanner for LLMs and LLM-backed applications [1] | Probes generate interactions, detectors decide whether a failure occurred, generators connect to the system under test, harnesses structure the run [1] | JSONL report per run with an entry per attempt, plus a hit log of successful attempts [1] |
| PyRIT (Microsoft) | Framework for assessing security and safety of generative AI systems, for automated and human-led red teaming [2] | Targets, converters, scorers and attacks, with memory that records conversations, scores and results [2] | Stored conversations and scores you can export and share [2] |
| promptfoo | Application-level red teaming and evaluation driven by a config file [3] | A purpose statement, plugins that generate adversarial inputs per risk area, strategies that deliver them, and targets [3] | A risk report with mitigation suggestions [3]; JSON, HTML or JUnit output in CI [8] |
| Inspect (UK AI Security Institute and Meridian Labs) | Framework for building LLM evaluations [4] | Datasets, solvers and scorers form a task; tools, agents and sandboxes support agentic evaluations [4] | Scores per sample for the scenarios you define [4] |
Pick by the question you are answering
- "What known failure classes does this endpoint have?" A scanner such as garak sweeps many probe families quickly against a model or REST endpoint. Good for a baseline and for comparing model versions.
- "Can a determined attacker get there over several turns?" PyRIT is built for orchestrated campaigns, including multi-turn strategies, with scoring and a record of every exchange [2].
- "Does our application still hold after this change?" promptfoo generates tests from a description of your app and maps plugin bundles to frameworks, with presets for the OWASP LLM Top 10, MITRE ATLAS and NIST AI RMF measures [6]. Its config lives next to the code, which suits CI.
- "Does our agent resist this scenario we wrote ourselves?" Inspect lets you define the dataset, the agent loop and the scorer, and run untrusted actions in a sandbox [4]. NIST's agent hijacking work added Inspect integration to AgentDojo for this reason [7].
What automated results cannot tell you
- A clean scan is not a clean system. Probes cover published failure classes. They know nothing about your tools, data or permissions unless you describe them, and adaptive attackers look for what is not in the set.
- Scorers make mistakes. Many detectors and graders are themselves models. Sample and check positives and negatives by hand before reporting a rate.
- Generic probes measure the model, not the impact. A probe that gets the model to reveal its system prompt is a finding of a different order from an injected document that triggers a tool call. Map results back to the threat model before triage.
- Coverage is not depth. Microsoft notes automation helps cover more of the risk landscape, while human expertise, context and judgement find what automation misses [5].
In practice teams combine them: a scanner for baselines when a model changes, an application-level suite in CI, and an orchestration or evaluation framework for the scenarios that matter most to the business. The next lesson covers turning their output into findings someone will fix.
Try it on ThreatDeck
- garak on the radar Check maintenance status and related tools before adding it to a pipeline.
- PyRIT on the radar Compare it with other orchestration frameworks for multi-turn campaigns.
- promptfoo on the radar See its ring and the playbooks that use it for CI regression suites.
- Inspect on the radar Review it as the base for custom agent evaluations.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
Probes create the interactions, generators connect to the target, and detectors classify the output as a hit or not. Because detectors can be wrong, hits need human review.
-
CI needs a repeatable test of the application with machine-readable results. promptfoo documents JSON, HTML and JUnit outputs and failing the build on test failures, which suits that job.
-
Automated results describe the probes that ran and the detectors that scored them. They do not cover your specific tools and data, novel techniques or attacks adapted to your defences.
-
Inspect's task model is a dataset, a solver (anything from one model call to a multi-turn agent) and a scorer, with sandboxing for untrusted code. That makes it a base for custom evaluations rather than a ready-made scan.
References
- garak: the LLM vulnerability scanner. NVIDIA (GitHub).
- PyRIT documentation. Microsoft.
- LLM red teaming guide. promptfoo.
- Inspect: an open-source framework for large language model evaluations. UK AI Security Institute.
- 3 takeaways from red teaming 100 generative AI products. Microsoft Security Blog, 2025-01-13.
- Red team configuration. promptfoo.
- Technical Blog: Strengthening AI Agent Hijacking Evaluations. NIST (US AI Safety Institute), 2025-01.
- CI/CD integration for LLM evaluation and security. promptfoo.