threatdeck
Live 13:45 UTC ·

Learn / Securing LLM applications

Lesson 2 of 69 minReviewed

Model output is untrusted input: handling it safely

Find every place model output is rendered, executed or forwarded, and apply the same controls you would to user input: encoding, parameterisation, allowlists and sandboxing.

A model's answer is shaped by everything in its context, including text written by outsiders. When that answer is passed to a browser, a database or a shell without validation, the application has an old-fashioned injection bug with a new delivery mechanism. OWASP files this as Improper Output Handling (LLM05) and lists the familiar outcomes: output passed to exec or a shell enabling remote code execution, generated JavaScript or Markdown causing XSS, unparameterised SQL, unsanitised file paths and unescaped email templates [1].

PortSwigger's research makes the indirect angle concrete: a hidden prompt on a web page can make an assistant return an XSS payload to the user who asked it to describe that page [2]. The user is authenticated, the model is behaving as designed, and the bug is in the renderer.

Sinks and their controls

Where model output goes and what to put in front of it
SinkWhat goes wrongDeterministic control
HTML or Markdown rendererXSS; auto-loading images that leak dataContext-aware output encoding [6], strict CSP, image and link allowlists
SQLInjection, destructive queriesParameterised queries, a read-only role scoped to the feature [1]
Shell, eval, code runnersRemote code executionNo eval; fixed commands with argument arrays; a sandbox with no credentials
HTTP fetcherSSRF to internal services and cloud metadataDestination allowlist, block private ranges, egress proxy [3]
File systemPath traversalCanonicalise, then check against a fixed root
Email or chat templatesPhishing content under your brandEscape output; human review before sending

Markdown images as an exfiltration channel

In April 2023 Johann Rehberger reported to Microsoft that Bing Chat could be steered by text on a web page into writing a markdown image whose URL carried data from the conversation. The chat client rendered it as an image, the browser fetched the URL automatically, and the data reached the attacker's server without a click [4]. The fix was confirmed in June 2023; the researcher observed a Content Security Policy that limits images to Bing's own domains [4]. Microsoft now describes blocking this class of exfiltration deterministically, alongside probabilistic injection defences [5].

The lesson generalises. Any feature that renders model output with automatic network requests (images, link previews, iframes, auto-fetched URLs) is an outbound channel. If an injected instruction can choose the URL, it can choose where data goes.

A checklist for every LLM feature

  1. Treat the model like any other untrusted user and validate its responses before they reach backend functions, as OWASP recommends [1].
  2. Encode for the destination context: HTML encoding for web pages, parameters for SQL, argument arrays for processes [1, 6].
  3. Ask for structured output (JSON with a schema), validate it, and reject anything that does not parse instead of repairing it.
  4. Restrict outbound requests the renderer or agent can make to an allowlist of hosts you control [3].
  5. Log model outputs that fail validation; a burst of rejected output is often the first sign someone is probing [1].

Try it on ThreatDeck

Self-check

4 questions

Pick an answer to see whether it is right and why. Only your score is saved, in this browser.

  1. Your chat UI renders model output as Markdown and the assistant summarises arbitrary web pages. Which control stops image-based data exfiltration regardless of what the model writes?
  2. A feature turns natural-language questions into SQL. What is the most effective control?
  3. An agent fetches any URL the model proposes and returns the page. Which classic vulnerability does this create?
  4. Why is model output treated as untrusted even when every user is authenticated?

References

  1. LLM05:2025 Improper Output Handling. OWASP GenAI Security Project, 2024-11.
  2. Web LLM attacks. PortSwigger Web Security Academy.
  3. Server Side Request Forgery Prevention Cheat Sheet. OWASP Cheat Sheet Series.
  4. Bing Chat: Data Exfiltration Exploit Explained. Johann Rehberger, Embrace The Red, 2023-06-18.
  5. How Microsoft defends against indirect prompt injection attacks. Microsoft Security Response Center, 2025-07-29.
  6. Cross Site Scripting Prevention Cheat Sheet. OWASP Cheat Sheet Series.
Completing the self-check also marks the lesson complete. Saved in this browser only.

Following

Follow the vendors, products, AI tools, techniques and actors you care about. Matching stories, CVEs and exploited bugs are marked, a Following tab appears in the feed, and returning visits show what changed. Saved only in this browser.

Presets

Get updates

A personal RSS feed of new stories on everything you follow. Paste it into any feed reader, Slack or Teams RSS app.