threatdeck
Live 13:45 UTC ·

Learn / Securing LLM applications

Lesson 6 of 69 minReviewed

Testing LLM applications in CI

Write security regression tests that run on every prompt, model, corpus or tool change, assert invariants rather than exact wording, and know what a passing suite does not prove.

An LLM feature can regress without a single line of application code changing. A new model version, an edited system prompt, a re-indexed corpus or one more tool in the list can each reopen an injection you fixed last month. That makes LLM security a good fit for the same discipline as unit tests: a fixed suite of cases that runs on every relevant change and fails the build when an invariant breaks.

Assert invariants, not wording

Model output varies from run to run, so exact-match tests are brittle. Security properties are usually expressible as things that must never appear or never happen. The open-source evaluation tool promptfoo, for example, supports deterministic assertions (text and regex matching, JSON validity, refusal detection, custom JavaScript or Python, cost and latency limits), checks on tool calls and agent trajectories, and model-graded rubrics; any assertion can be negated with a not- prefix and weighted [1].

Security invariants and how to assert them
InvariantTest inputAssertion
No data leaves through rendered outputA fixture page or document carrying hidden instructionsnot-regex for images or links to hosts outside your allowlist
System prompt and context do not leakExtraction attemptsnot-contains a canary string planted in the system prompt
No unexpected actionsInjected documents in a summarise taskTrajectory or tool-call assertions that the send or write tool was not used
Out-of-scope requests are refusedRequests outside the feature's jobis-refusal or a narrow rubric
Downstream code gets valid dataNormal and adversarial inputsis-json plus schema checks
promptfoo test case asserting two invariants (illustrative)
tests:
  - description: Summary of a page with hidden instructions
    vars:
      doc: file://fixtures/page-with-hidden-instructions.md
    assert:
      - type: not-regex
        value: '!\[[^\]]*\]\(https?://(?!assets\.example\.com)'
      - type: not-contains
        value: CANARY-7f3a

Keep the fixtures in the repository next to the prompts they protect. Every confirmed finding from a red-team exercise or incident becomes a new case, so the same bug cannot quietly return.

Wiring it into the pipeline

  • Run promptfoo eval -c promptfooconfig.yaml in the job, write JSON, HTML and JUnit reports with repeated -o flags, and use --fail-on-error or a small script that fails below a pass-rate threshold [2].
  • Trigger the job on changes to prompts, model identifiers, retrieval configuration and tool definitions, not only on code.
  • Cache responses between runs (PROMPTFOO_CACHE_PATH, PROMPTFOO_CACHE_TTL) to control cost and rate limits, and tag runs with the commit for traceability [2].
  • Schedule broader generated probes, such as promptfoo redteam run [2] or garak, NVIDIA's open-source scanner for prompt injection, data leakage, jailbreaks and other failure classes [3], outside the merge path where their run time and noise do not block developers.

What a green build does not prove

A passing suite shows that the cases you wrote behave as expected today. It is not evidence of resistance to an attacker who adapts: adaptive attacks have bypassed most of a set of twelve published defences that looked strong on fixed test sets [4]. Use CI to stop known regressions and to check invariants that your deterministic controls are meant to guarantee, and keep production monitoring for the rest; the NCSC recommends logging inputs, outputs, tool use and failed calls so that probing is visible early [5].

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. Which change should trigger the LLM security test suite?
  2. What is a canary string in the system prompt used for?
  3. Your suite passes at 100%. What can you conclude?
  4. How do you test that a summariser never emails anyone when a document contains injected instructions?

References

  1. Assertions and metrics. promptfoo documentation.
  2. CI/CD integration. promptfoo documentation.
  3. garak: LLM vulnerability scanner. NVIDIA (GitHub).
  4. The Attacker Moves Second: Stronger Adaptive Attacks Bypass Defenses against LLM Jailbreaks and Prompt Injections. Nasr et al. (arXiv:2510.09023), 2025-10.
  5. Prompt injection is not SQL injection (it may be worse). UK National Cyber Security Centre, 2025-12-08.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.