Learn / Securing LLM applications
Lesson 6 of 69 minReviewed
Testing LLM applications in CI
Write security regression tests that run on every prompt, model, corpus or tool change, assert invariants rather than exact wording, and know what a passing suite does not prove.
An LLM feature can regress without a single line of application code changing. A new model version, an edited system prompt, a re-indexed corpus or one more tool in the list can each reopen an injection you fixed last month. That makes LLM security a good fit for the same discipline as unit tests: a fixed suite of cases that runs on every relevant change and fails the build when an invariant breaks.
Assert invariants, not wording
Model output varies from run to run, so exact-match tests are brittle. Security properties are usually expressible as things that must never appear or never happen. The open-source evaluation tool promptfoo, for example, supports deterministic assertions (text and regex matching, JSON validity, refusal detection, custom JavaScript or Python, cost and latency limits), checks on tool calls and agent trajectories, and model-graded rubrics; any assertion can be negated with a not- prefix and weighted [1].
| Invariant | Test input | Assertion |
|---|---|---|
| No data leaves through rendered output | A fixture page or document carrying hidden instructions | not-regex for images or links to hosts outside your allowlist |
| System prompt and context do not leak | Extraction attempts | not-contains a canary string planted in the system prompt |
| No unexpected actions | Injected documents in a summarise task | Trajectory or tool-call assertions that the send or write tool was not used |
| Out-of-scope requests are refused | Requests outside the feature's job | is-refusal or a narrow rubric |
| Downstream code gets valid data | Normal and adversarial inputs | is-json plus schema checks |
tests:
- description: Summary of a page with hidden instructions
vars:
doc: file://fixtures/page-with-hidden-instructions.md
assert:
- type: not-regex
value: '!\[[^\]]*\]\(https?://(?!assets\.example\.com)'
- type: not-contains
value: CANARY-7f3aKeep the fixtures in the repository next to the prompts they protect. Every confirmed finding from a red-team exercise or incident becomes a new case, so the same bug cannot quietly return.
Wiring it into the pipeline
- Run
promptfoo eval -c promptfooconfig.yamlin the job, write JSON, HTML and JUnit reports with repeated-oflags, and use--fail-on-erroror a small script that fails below a pass-rate threshold [2]. - Trigger the job on changes to prompts, model identifiers, retrieval configuration and tool definitions, not only on code.
- Cache responses between runs (
PROMPTFOO_CACHE_PATH,PROMPTFOO_CACHE_TTL) to control cost and rate limits, and tag runs with the commit for traceability [2]. - Schedule broader generated probes, such as
promptfoo redteam run[2] or garak, NVIDIA's open-source scanner for prompt injection, data leakage, jailbreaks and other failure classes [3], outside the merge path where their run time and noise do not block developers.
What a green build does not prove
A passing suite shows that the cases you wrote behave as expected today. It is not evidence of resistance to an attacker who adapts: adaptive attacks have bypassed most of a set of twelve published defences that looked strong on fixed test sets [4]. Use CI to stop known regressions and to check invariants that your deterministic controls are meant to guarantee, and keep production monitoring for the rest; the NCSC recommends logging inputs, outputs, tool use and failed calls so that probing is visible early [5].
Try it on ThreatDeck
- promptfoo on the radar Check activity and alternatives before standardising on an evaluation tool.
- garak on the radar Schedule probe sweeps for injection, leakage and jailbreak classes.
- Red-team an LLM application Turn red-team findings into regression cases for the suite.
- Coverage Record which techniques each system is tested against.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
Each of these can change model behaviour and reopen a fixed issue without any code change, so the job should watch all of them.
-
A unique, meaningless token that should never appear in output makes leakage trivially testable, whatever wording the model uses.
-
Tests cover known cases. Research on adaptive attacks shows defences that pass fixed suites can still fall to tuned attacks, so monitoring and deterministic controls remain necessary.
-
The invariant is about actions, so assert on the actions. Asking the model is unreliable and exact-match comparisons break on harmless wording changes.
References
- Assertions and metrics. promptfoo documentation.
- CI/CD integration. promptfoo documentation.
- garak: LLM vulnerability scanner. NVIDIA (GitHub).
- The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses against LLM Jailbreaks and Prompt Injections. Nasr et al. (arXiv:2510.09023), 2025-10.
- Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.