Learn / Securing LLM applications
Lesson 2 of 69 minReviewed
Model output is untrusted input: handling it safely
Find every place model output is rendered, executed or forwarded, and apply the same controls you would to user input: encoding, parameterisation, allowlists and sandboxing.
A model's answer is shaped by everything in its context, including text written by outsiders. When that answer is passed to a browser, a database or a shell without validation, the application has an old-fashioned injection bug with a new delivery mechanism. OWASP files this as Improper Output Handling (LLM05) and lists the familiar outcomes: output passed to exec or a shell enabling remote code execution, generated JavaScript or Markdown causing XSS, unparameterised SQL, unsanitised file paths and unescaped email templates [1].
PortSwigger's research makes the indirect angle concrete: a hidden prompt on a web page can make an assistant return an XSS payload to the user who asked it to describe that page [2]. The user is authenticated, the model is behaving as designed, and the bug is in the renderer.
Sinks and their controls
| Sink | What goes wrong | Deterministic control |
|---|---|---|
| HTML or Markdown renderer | XSS; auto-loading images that leak data | Context-aware output encoding [6], strict CSP, image and link allowlists |
| SQL | Injection, destructive queries | Parameterised queries, a read-only role scoped to the feature [1] |
Shell, eval, code runners | Remote code execution | No eval; fixed commands with argument arrays; a sandbox with no credentials |
| HTTP fetcher | SSRF to internal services and cloud metadata | Destination allowlist, block private ranges, egress proxy [3] |
| File system | Path traversal | Canonicalise, then check against a fixed root |
| Email or chat templates | Phishing content under your brand | Escape output; human review before sending |
Markdown images as an exfiltration channel
In April 2023 Johann Rehberger reported to Microsoft that Bing Chat could be steered by text on a web page into writing a markdown image whose URL carried data from the conversation. The chat client rendered it as an image, the browser fetched the URL automatically, and the data reached the attacker's server without a click [4]. The fix was confirmed in June 2023; the researcher observed a Content Security Policy that limits images to Bing's own domains [4]. Microsoft now describes blocking this class of exfiltration deterministically, alongside probabilistic injection defences [5].
The lesson generalises. Any feature that renders model output with automatic network requests (images, link previews, iframes, auto-fetched URLs) is an outbound channel. If an injected instruction can choose the URL, it can choose where data goes.
A checklist for every LLM feature
- Treat the model like any other untrusted user and validate its responses before they reach backend functions, as OWASP recommends [1].
- Encode for the destination context: HTML encoding for web pages, parameters for SQL, argument arrays for processes [1, 6].
- Ask for structured output (JSON with a schema), validate it, and reject anything that does not parse instead of repairing it.
- Restrict outbound requests the renderer or agent can make to an allowlist of hosts you control [3].
- Log model outputs that fail validation; a burst of rejected output is often the first sign someone is probing [1].
Try it on ThreatDeck
- Detection: markdown image exfiltration Hunt for model output that embeds images or reference links to external hosts.
- Improper Output Handling (LLM05) Read recent stories where output handling turned injection into impact.
- LLM Response Rendering (ATLAS AML.T0077) See how ATLAS models abuse of the rendering step.
- Labs & practice PortSwigger's Web LLM attacks labs include insecure output handling exercises.
Self-check
4 questionsPick an answer to see whether it is right and why. Only your score is saved, in this browser.
-
The exfiltration happens when the browser fetches an attacker-chosen URL. Restricting which hosts can be loaded is deterministic; prompt instructions and input classifiers only lower the odds that the model writes the image.
-
Least privilege on the database bounds what any generated query can do, whatever the model was persuaded to write. Keyword filters and instructions are easy to evade.
-
If injected text can choose the URL, the server makes requests on the attacker's behalf, including to internal addresses. Allowlists and blocking private ranges are the standard SSRF controls.
-
The output reflects every writer of the context, including authors of documents and pages the app retrieved. Authentication says who asked, not who influenced the answer.
References
- LLM05:2025 Improper Output Handling. OWASP GenAI Security Project, 2024-11.
- Web LLM attacks. PortSwigger Web Security Academy.
- Server Side Request Forgery Prevention Cheat Sheet. OWASP Cheat Sheet Series.
- Bing Chat: Data Exfiltration Exploit Explained. Johann Rehberger, Embrace The Red, 2023-06-18.
- How Microsoft defends against indirect prompt injection attacks. Microsoft Security Response Center, 2025-07-29.
- Cross Site Scripting Prevention Cheat Sheet. OWASP Cheat Sheet Series.