Improper Output Handling in LLM Applications
detect, understand, remediate

Rendered as HTML in the browser

The application receives model text and writes it into the page through innerHTML, dangerouslySetInnerHTML, a templating engine that does not escape, a markdown renderer that allows raw HTML, or a chat UI that auto-links URLs. Any tag, script, image, iframe, or autolinked URI the model produced fires in the user's browser.

Concatenated into a SQL or NoSQL query

A summarisation, classification, or SQL-generation step returns text that the application interpolates into a query. The model can return UNION clauses, comment markers, JSON path operators, NoSQL operators, or full statements that the parser executes.

Used as a URL in a server-side fetch

An agent or RAG flow asks the model for a URL to fetch, a hostname to call, or a redirect target. Without an allow-list the application contacts attacker-controlled infrastructure or internal cloud metadata endpoints; this is the LLM expression of server-side request forgery.

Executed in a shell, container, or interpreter

Code-execution agents, data-analysis sandboxes, and notebook-runner backends pass model output to a shell or Python interpreter. Even sandboxed runners often share a network namespace, a filesystem mount, or a credential the attacker can reach through the executed code.

Written to a file or downloaded to disk

A report-generation flow writes a model-produced filename or path. The model returns ../../etc/hosts, an alternate data stream, a Windows reserved name, or a path containing null bytes; the surrounding code happily writes outside the intended directory.

Passed as an argument to an agent tool

A tool-using agent calls a registered function: send_email(to, body), update_record(id, fields), charge_card(amount, currency), invite_user(email, role). The arguments come from the model. Without schema validation, allow-listed values, and per-tool authorisation, the call executes with the model's judgement rather than the application's.

Rendered as Markdown with active content

A markdown renderer that allows raw HTML, image tags, autolinks, or click-tracking link shorteners turns benign-looking model output into a stored exfiltration channel. The image tag fetches a logging URL with the conversation context in the query string.

Inserted into a downstream system prompt or memory

Multi-turn agents store earlier model responses into long-term memory, then use that memory as part of the system prompt for later requests. An earlier output containing a smuggled instruction becomes a permanent injection for every later session that retrieves the same memory.

Treating model output as already sanitised

Engineering teams that would never accept user-supplied HTML write a feature that pipes model output directly into a rendered surface. The mental model is "the model wrote it, so it is safe". Every byte of model output is untrusted input; that is the rule that fails most often.

String concatenation instead of parameterisation

Model output is interpolated into SQL, shell, regular expressions, or external API bodies as a string. The downstream layer parses the string the same way it parses an attacker-supplied payload, because that is what model output structurally is.

No schema validation on agent tool arguments

A registered tool accepts arbitrary arguments because the application trusts the model to format them. The model produces values outside the intended range, type, or set: a delete_user call with the wrong id, a charge_card with the wrong currency, a send_message with the wrong recipient.

Markdown or HTML renderer accepts raw tags

A "rich" output renderer permits raw HTML, image tags, autolinks, embedded videos, or click-tracking shorteners. The renderer was chosen for product polish and not for the security posture. The polish surface is the exfiltration surface.

No allow-list on outbound URLs

An agent fetches whatever URL the model suggests. The fetcher has no allow-list, no DNS rebinding protection, no link-local block, and no internal-network deny rule. The first SSRF lands within a session.

Tool authorisation reads the model rather than the session

The tool checks whether the model "intends" the user to perform the action by re-asking the model. The check is recursive on the same untrusted source. The application's authorisation layer must read the user identity, role, scope, and policy rather than the model's self-reported reason for the call.

Automated detection

• SecPortal's code scanning runs against connected repositories and flags model-call patterns where the response is concatenated into a SQL query, written into a rendered DOM, passed to a shell, used as a fetch target, or fed as the argument vector of a tool registration without a validating schema
• Authenticated scanning probes the LLM-backed endpoint with output-shaping prompts that ask the model to emit candidate payloads for the downstream sink, observes whether the surrounding application renders, executes, or fetches the result, and records the request, response, rendered output, and downstream side effects as evidence on the finding
• External scanning discovers exposed LLM-backed endpoints, public agent webhooks, and tool-callback URLs reachable from the public internet that may render or execute model output without authentication
• Continuous monitoring re-runs the output-handling probe on a defined cadence so a renderer upgrade, a model upgrade, a new tool registration, or a switched markdown library is caught against the previous baseline

Manual testing

• Ask the model to emit a deliberately structured payload for the downstream sink: an image tag with a logging URL for HTML rendering, a UNION clause for a SQL builder, a metadata-endpoint URL for an agent fetcher, a path with traversal segments for a file writer
• For agent tools, enumerate every registered tool and craft a prompt that asks the model to call each tool with out-of-policy arguments; record which calls fire and what the application does with the result
• Read the rendering and post-processing layer end to end: identify every place the model output enters a sink, then build a regression probe that ships a benign canary payload through each path
• Submit the same probe through multi-turn conversation memory, long-term agent memory, and shared transcripts to confirm whether stored output handling is in scope
• Inspect the network log for the test session to confirm that no model-produced URL was fetched without an allow-list pass and that no DNS lookup resolves to a link-local or internal-network address

Finding record with output, sink, and side effect as evidence

The finding captures the original prompt, the model response, the downstream sink that consumed the response (HTML rendering, SQL query, fetch target, shell argument, tool call, file write), and the observed side effect (script execution in the test browser, query log entry, outbound request, file write outside the intended directory). The evidence is what the engineering team needs to reproduce the attack against the same surrounding code path.

Code scanning across LLM call sites

Code scanning runs against connected GitHub, GitLab, and Bitbucket repositories. Findings surface where model output flows into a query builder, a templating engine, a subprocess call, an http client, a file system write, or an unbounded tool argument vector. The remediation lands at the call site rather than at a perimeter filter.

Authenticated scanning with output-shaping payloads

Authenticated scanning runs against the LLM-backed endpoint with a curated set of output-shaping prompts under a real session. Each probe records whether the response renders, executes, or fetches inside the surrounding application, so the finding ties to a downstream sink rather than to the model's text alone.

Continuous monitoring against renderer and model drift

Continuous monitoring re-runs the output-handling probe on the configured cadence. A model upgrade, a markdown library swap, or a new tool registration that re-opens a previously closed finding shows up against the baseline rather than waiting for the next pentest cycle.

Retest after the remediation ships

Once the fix deploys, a targeted retest replays the original payload through the new sink and records the post-fix response on the finding. The finding closes against the evidence rather than against a developer's assertion that the renderer or schema is now safe.

AI-assisted writeups with explicit honest scope

AI reports generate the writeup, the executive summary, and the developer-facing reproduction steps from the finding record. The narrative stays within the verified evidence on the finding (the prompt, the response, the sink, the observed side effect) and does not invent renderer behaviour or runtime guardrails the product does not have.

Finding overrides for documented exceptions

Where a downstream sink is a deliberate test fixture or a sanctioned internal-only surface, finding overrides record the suppression rationale, the owner, and the expiry on the finding itself. The exception lives on the operating record rather than in a parallel spreadsheet.

Compliance tracking pairs the fix to control evidence

Compliance tracking maps improper-output-handling findings to the controls that read against them (ISO 27001 A.8.28 secure coding, SOC 2 CC6.1 logical access, NIST SSDF PW.5 secure coding practices, NIST AI RMF Map and Measure, ISO/IEC 42001 AI system lifecycle). The same finding feeds the engineering ticket and the auditor evidence pack.