threatdeck
Live 13:45 UTC ·

Learn / Securing LLM applications

Lesson 4 of 68 minReviewed

Sensitive data and system prompt leakage

Assume anything in the context can come back out, keep secrets and authorisation logic out of prompts, and minimise what data reaches the model in the first place.

Sensitive data reaches a model through more doors than the chat box: the system prompt, retrieved documents, tool results, conversation history and any fine-tuning data. OWASP's Sensitive Information Disclosure entry (LLM02) gives the common outcomes as leaked personal data, exposed proprietary algorithms and confidential business information appearing in answers [1]. NIST's Generative AI Profile lists data privacy among the twelve risks it asks organisations to manage [5].

The system prompt is not a secret

OWASP's System Prompt Leakage entry (LLM07) is direct: the system prompt should not be considered a secret, nor used as a security control [2]. The risk is not that someone reads your persona instructions. It is what the prompt reveals or replaces: credentials and connection strings, internal decision rules such as transaction limits, the criteria a filter uses to reject requests, and role structures that point an attacker at privilege escalation [2].

Treat leakage as a given: the injection techniques that override instructions can also make the model repeat them, which is why OWASP recommends not relying on the prompt for strict behaviour control [2]. If reading the prompt would help an attacker, the design is relying on obscurity.

What not to put in a prompt, and where it belongs
Never in the promptWhyPut it here instead
API keys, tokens, database credentialsAnything in context can be echoed or exfiltratedA secret manager, used by the tool server or backend that makes the call [2]
Authorisation rules ("only admins may…")Injected text can override instructionsDeterministic checks in the API the tool calls [2]
Business limits (refund caps, approval thresholds)Leaks the rule and lets an attacker probe around itBackend validation with server-side approval workflows
Other users' recordsThe model cannot keep one user's data from anotherRetrieval filtered by the caller's identity [3]
Filter and moderation criteriaTells an attacker exactly what to avoidGuardrail configuration outside the context [2]

Minimise what the model can reach

The most reliable way to prevent disclosure is to keep data out of the context. OWASP's mitigations start with least-privilege access to data sources, sanitising or masking sensitive content, and tokenising or redacting patterns before processing [1]. PortSwigger adds two rules worth adopting verbatim as design reviews: treat every API the model can call as if it were public, and only give the model data the lowest-privileged user of the feature may see [3].

  • Pass identifiers, not records, and let tools fetch only the fields a step needs.
  • Redact secrets and personal data from tool output before it is appended to the context.
  • Limit users' ability to view or override system-level configuration, and keep error messages free of internal detail [1].

Logs and transcripts are copies too

Prompt and response logs, traces and evaluation datasets contain whatever the context contained. Classify them like the source data and set retention deliberately. Client-side tools keep copies as well: Claude Code, for example, stores session transcripts in plaintext under ~/.claude/projects/ for 30 days by default, adjustable with cleanupPeriodDays [4]. Path 4 of this hub covers those local stores for AI coding assistants.

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. A support assistant's system prompt says "refunds above $500 need manager approval" and the refund tool takes any amount. What is the right fix?
  2. Where should the API key a tool uses be kept?
  3. What is OWASP's position on system prompts in LLM07?
  4. Which measure most reduces the chance an assistant discloses another user's data?

References

  1. LLM02:2025 Sensitive Information Disclosure. OWASP GenAI Security Project, 2024-11.
  2. LLM07:2025 System Prompt Leakage. OWASP GenAI Security Project, 2024-11.
  3. Web LLM attacks. PortSwigger Web Security Academy.
  4. Claude Code: Data usage. Anthropic.
  5. NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.