threatdeck
Live 13:45 UTC ·

Learn / Red teaming AI systems

Lesson 6 of 68 minReviewed

Running it continuously: regression suites and retest triggers

Turn findings into a regression suite, decide which changes trigger a retest, gate CI on results without flaky failures, and keep periodic human rounds for what automation misses.

An AI system changes more often than its code. A provider ships a new model version, someone edits the system prompt, a team connects another MCP server, a new document source is indexed. Each of these can undo a fix without anyone touching the code that was reviewed. Microsoft's red team states the consequence as one of its lessons: the work of securing AI systems will never be complete [1], and it describes AI red teaming as a continuous process that must adapt as risks change [2].

What should trigger a retest

Changes and the tests they should run
ChangeWhy it mattersRun
Model or model versionInstruction-following and refusal behaviour shift between versionsFull regression suite plus a baseline scan
System prompt or orchestrationGuardrail wording and context assembly change what injected text competes withRegression suite
New tool, MCP server or permissionNew impact: the threat model changedRescope, then targeted human testing and new cases
New retrieval sourceNew writers can reach the contextIndirect injection cases against that source
Guardrail or filter updateDefences are often bypassed by attacks adapted to them [3]Regression suite plus adaptive variants

Build the suite from findings

Every finding from the last lesson should leave a case: the attacker-controlled input placed where it arrived in the original test, and an assertion on the impact, for example that the canary never reaches an outside recipient or that the send tool was not called. Assert on actions and tool calls where you can; assertions on wording break with every model update and miss the impact that mattered.

promptfoo documents the CI pattern: run evaluations on pull requests that touch prompts or configuration, run red-team scans on a schedule, fail the build on test failures or below a pass-rate threshold, and publish JSON, HTML or JUnit results [4]:

CI steps (promptfoo CLI, from its CI documentation)
# on pull requests that change prompts, tools or config
npx promptfoo@latest eval -c promptfooconfig.yaml --fail-on-error -o results.junit.xml

# on a schedule, against a staging deployment
npx promptfoo@latest redteam run

Gate without flakiness

  • Repeat high-impact cases and gate on a rate, not a single run. NIST found repeated attempts changed per-task success rates significantly [5]; one passing run proves little.
  • Separate hard gates from trend lines. A case that once exfiltrated data should block a release if it succeeds even once. Broad scan scores belong on a dashboard, not in the gate.
  • Cache and cap cost. Cache responses between runs and lower concurrency to stay inside rate limits [4]; set a token budget per pipeline run.
  • Version the suite with the system. Store cases next to the prompts and tool definitions they test, and review changes to both together.

Keep humans in the rotation

Automated suites protect against known failures coming back. They do not find new ones, and adaptive attacks keep getting stronger [3, 5]. Schedule human rounds when the threat model changes and at a fixed interval otherwise, and feed every new finding back into the suite. NIST AI 600-1 notes that red teaming can happen before or after a system is made available to the public, and lists it among the actions of the AI RMF's Measure function [6].

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. Which change most clearly requires rescoping and new human testing, not only rerunning the suite?
  2. What should a regression case for an exfiltration finding assert?
  3. A high-impact case passes once in CI. Why is that weak evidence?
  4. Why keep periodic human red team rounds when a CI suite exists?

References

  1. Lessons From Red Teaming 100 Generative AI Products. Bullwinkel et al., Microsoft (arXiv:2501.07238), 2025-01.
  2. 3 takeaways from red teaming 100 generative AI products. Microsoft Security Blog, 2025-01-13.
  3. The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses against LLM Jailbreaks and Prompt Injections. Nasr et al. (arXiv:2510.09023), 2025-10.
  4. CI/CD integration for LLM evaluation and security. promptfoo.
  5. Technical Blog: Strengthening AI Agent Hijacking Evaluations. NIST (US AI Safety Institute), 2025-01.
  6. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.