Learn / Securing LLM applications
Lesson 4 of 68 minReviewed
Sensitive data and system prompt leakage
Assume anything in the context can come back out, keep secrets and authorisation logic out of prompts, and minimise what data reaches the model in the first place.
Sensitive data reaches a model through more doors than the chat box: the system prompt, retrieved documents, tool results, conversation history and any fine-tuning data. OWASP's Sensitive Information Disclosure entry (LLM02) gives the common outcomes as leaked personal data, exposed proprietary algorithms and confidential business information appearing in answers [1]. NIST's Generative AI Profile lists data privacy among the twelve risks it asks organisations to manage [5].
The system prompt is not a secret
OWASP's System Prompt Leakage entry (LLM07) is direct: the system prompt should not be considered a secret, nor used as a security control [2]. The risk is not that someone reads your persona instructions. It is what the prompt reveals or replaces: credentials and connection strings, internal decision rules such as transaction limits, the criteria a filter uses to reject requests, and role structures that point an attacker at privilege escalation [2].
Treat leakage as a given: the injection techniques that override instructions can also make the model repeat them, which is why OWASP recommends not relying on the prompt for strict behaviour control [2]. If reading the prompt would help an attacker, the design is relying on obscurity.
| Never in the prompt | Why | Put it here instead |
|---|---|---|
| API keys, tokens, database credentials | Anything in context can be echoed or exfiltrated | A secret manager, used by the tool server or backend that makes the call [2] |
| Authorisation rules ("only admins may…") | Injected text can override instructions | Deterministic checks in the API the tool calls [2] |
| Business limits (refund caps, approval thresholds) | Leaks the rule and lets an attacker probe around it | Backend validation with server-side approval workflows |
| Other users' records | The model cannot keep one user's data from another | Retrieval filtered by the caller's identity [3] |
| Filter and moderation criteria | Tells an attacker exactly what to avoid | Guardrail configuration outside the context [2] |
Minimise what the model can reach
The most reliable way to prevent disclosure is to keep data out of the context. OWASP's mitigations start with least-privilege access to data sources, sanitising or masking sensitive content, and tokenising or redacting patterns before processing [1]. PortSwigger adds two rules worth adopting verbatim as design reviews: treat every API the model can call as if it were public, and only give the model data the lowest-privileged user of the feature may see [3].
- Pass identifiers, not records, and let tools fetch only the fields a step needs.
- Redact secrets and personal data from tool output before it is appended to the context.
- Limit users' ability to view or override system-level configuration, and keep error messages free of internal detail [1].
Logs and transcripts are copies too
Prompt and response logs, traces and evaluation datasets contain whatever the context contained. Classify them like the source data and set retention deliberately. Client-side tools keep copies as well: Claude Code, for example, stores session transcripts in plaintext under ~/.claude/projects/ for 30 days by default, adjustable with cleanupPeriodDays [4]. Path 4 of this hub covers those local stores for AI coding assistants.
Try it on ThreatDeck
- System Prompt Leakage (LLM07) Read recent leakage stories and what the leaked prompts exposed.
- Sensitive Information Disclosure (LLM02) Follow disclosure incidents and the defences mapped to them.
- Claude Code data-exposure sheet See where a coding assistant keeps transcripts and credentials on disk.
- Get visibility into Shadow AI Find where sensitive data is already flowing to unapproved AI services.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
A rule that exists only in the prompt can be leaked and overridden by injected text. Business limits must be enforced by the backend that executes the action.
-
Anything in context can be echoed or exfiltrated. The model only needs to request an action; the component that performs it should hold the credential.
-
OWASP says the real risk is what the prompt reveals or replaces: credentials, rules and roles. Security has to be enforced independently of the prompt.
-
Data that never enters the context cannot be disclosed. Scoping retrieval and tool results to the caller is a deterministic control; prompt instructions are not.
References
- LLM02:2025 Sensitive Information Disclosure. OWASP GenAI Security Project, 2024-11.
- LLM07:2025 System Prompt Leakage. OWASP GenAI Security Project, 2024-11.
- Web LLM attacks. PortSwigger Web Security Academy.
- Claude Code: Data usage. Anthropic.
- NIST AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile. NIST, 2024-07.