Learn / Red teaming AI systems
Lesson 6 of 68 minReviewed
Running it continuously: regression suites and retest triggers
Turn findings into a regression suite, decide which changes trigger a retest, gate CI on results without flaky failures, and keep periodic human rounds for what automation misses.
An AI system changes more often than its code. A provider ships a new model version, someone edits the system prompt, a team connects another MCP server, a new document source is indexed. Each of these can undo a fix without anyone touching the code that was reviewed. Microsoft's red team states the consequence as one of its lessons: the work of securing AI systems will never be complete [1], and it describes AI red teaming as a continuous process that must adapt as risks change [2].
What should trigger a retest
| Change | Why it matters | Run |
|---|---|---|
| Model or model version | Instruction-following and refusal behaviour shift between versions | Full regression suite plus a baseline scan |
| System prompt or orchestration | Guardrail wording and context assembly change what injected text competes with | Regression suite |
| New tool, MCP server or permission | New impact: the threat model changed | Rescope, then targeted human testing and new cases |
| New retrieval source | New writers can reach the context | Indirect injection cases against that source |
| Guardrail or filter update | Defences are often bypassed by attacks adapted to them [3] | Regression suite plus adaptive variants |
Build the suite from findings
Every finding from the last lesson should leave a case: the attacker-controlled input placed where it arrived in the original test, and an assertion on the impact, for example that the canary never reaches an outside recipient or that the send tool was not called. Assert on actions and tool calls where you can; assertions on wording break with every model update and miss the impact that mattered.
promptfoo documents the CI pattern: run evaluations on pull requests that touch prompts or configuration, run red-team scans on a schedule, fail the build on test failures or below a pass-rate threshold, and publish JSON, HTML or JUnit results [4]:
# on pull requests that change prompts, tools or config
npx promptfoo@latest eval -c promptfooconfig.yaml --fail-on-error -o results.junit.xml
# on a schedule, against a staging deployment
npx promptfoo@latest redteam runGate without flakiness
- Repeat high-impact cases and gate on a rate, not a single run. NIST found repeated attempts changed per-task success rates significantly [5]; one passing run proves little.
- Separate hard gates from trend lines. A case that once exfiltrated data should block a release if it succeeds even once. Broad scan scores belong on a dashboard, not in the gate.
- Cache and cap cost. Cache responses between runs and lower concurrency to stay inside rate limits [4]; set a token budget per pipeline run.
- Version the suite with the system. Store cases next to the prompts and tool definitions they test, and review changes to both together.
Keep humans in the rotation
Automated suites protect against known failures coming back. They do not find new ones, and adaptive attacks keep getting stronger [3, 5]. Schedule human rounds when the threat model changes and at a fixed interval otherwise, and feed every new finding back into the suite. NIST AI 600-1 notes that red teaming can happen before or after a system is made available to the public, and lists it among the actions of the AI RMF's Measure function [6].
Try it on ThreatDeck
- promptfoo on the radar Check its status before adopting it as the CI runner for regression cases.
- Coverage Mark techniques with current regression tests for each system and spot gaps.
- Playbook: red-team an LLM application Use the checklist items on CI regression and retest triggers.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
A new tool adds a new impact, so the threat model changes. Existing cases cannot cover actions that did not exist when they were written.
-
Assert on the impact and tool calls. Refusal wording changes between model versions and can pass while the action still happens.
-
NIST's evaluations showed repeated attempts changed success rates substantially. Gate high-impact cases on repeated runs and treat any success as a failure.
-
Regression suites encode yesterday's findings. Adaptive attacks and new capabilities need human judgement, and each new finding then becomes a suite case.
References
- Lessons From Red Teaming 100 Generative AI Products. Bullwinkel et al., Microsoft (arXiv:2501.07238), 2025-01.
- 3 takeaways from red teaming 100 generative AI products. Microsoft Security Blog, 2025-01-13.
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses against LLM Jailbreaks and Prompt Injections. Nasr et al. (arXiv:2510.09023), 2025-10.
- CI/CD integration for LLM evaluation and security. promptfoo.
- Technical Blog: Strengthening AI Agent Hijacking Evaluations. NIST (US AI Safety Institute), 2025-01.
- NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.