New Technique Medium TD-2026-0031 For AppSec, AI Eng, CISO AI-assisted
CASE-Bench: repository context changes security of LLM-generated code (145 CVE tasks)
An arXiv preprint introduces CASE-Bench, 145 repository-level tasks built from real CVEs across 51 projects, each with functional tests and a validated proof-of-concept exploit. The authors tested 13 context construction methods and six controls with six LLMs. Functionality-oriented context raised secure pass rates by at most 3.7 pp, and withholding a safeguard lowered them by up to 8.1 pp.
Why it matters
Coding assistants and agents build repository context mostly for functional correctness, so they may miss existing authorization checks and copy insecure code. Organizations using these tools on security-sensitive code may ship vulnerable generated code that passes functional tests.
Do this
- Review how your coding assistants and agents select repository context, and confirm that files enforcing authorization and input validation are included for security-sensitive changes.
- Do not rely on functional test pass rates alone for AI-generated code; add exploit-based or security regression tests for changed endpoints.
- Audit existing repositories for insecure code patterns that could be imitated by AI tools, and fix or annotate them before they are used as context.
- Add repository-level secure-code evaluations to AI coding tool assessments, using tasks that include the project's own safeguards.
Signals
- NoCISA KEV
- 23/100Priority
- 60%Confidence
Evidence
“For security-sensitive code, this context decides whether the model sees the project's safeguards (e.g., authorization checks) or insecure code to imitate.”
“functionality-oriented context raises the functional pass rate by up to 8.2 percentage points (pp) but the secure pass rate by at most 3.7 pp”
“withholding a safeguard lowers the secure pass rate by up to 8.1 pp”
“coding agents observe the relevant safeguard in only 27-36% of trajectories”
Sources
Affected / related
CASE-Bench, repository context construction for LLM coding assistants
Published 2026-11-04 10:29 UTC · updated 2026-11-08 17:52 UTC · AI-assisted summary, checked against quoted sources · How we score