# Output handling

Treat model output as untrusted input: it can contain anything that was in the
context. Follow every output to the place where it ends up, including text
that tools write somewhere.

## HTML and templates

- Search templates and views for `|safe`, `Markup(`, `{% autoescape false %}`,
  `mark_safe`, `innerHTML` and `dangerouslySetInnerHTML`. Follow each one back
  and report it when the value comes from a model (LLM10).
- Markdown converted to HTML: many Markdown libraries pass raw HTML through by
  default or can be configured to. Rendered model output must be sanitised with
  an allow-list of tags (for example `nh3.clean`) before it is marked as safe.
- JSON inside a `<script>` tag (JSON-LD, initial state): `json.dumps` doesn't
  escape `</script>`. Use Jinja's `tojson` filter, or escape `<`, `>` and `&`.
- URLs from model output in `href` or `src` need a scheme check (no
  `javascript:`).
- Check escaping in every other context the output reaches: meta tags, email
  subjects and bodies, HTTP headers.

## Validation before publishing

- Find the code between the model and each destination. Output published,
  sent or executed without a person reviewing it first is autonomy (LLM03):
  report it unless the design says why it is needed.
- When there is no review, or the reviewer may be the attacker (for example
  the owner of the content being described), look for validation between the
  model and the destination: links, HTML, length, and any content the design
  doesn't allow.
- Checks that need judgement (invented facts, claims about qualifications)
  usually rely on a moderation model or classifier. Confirm that it runs on
  every output, including approved ones, and that a failure blocks publishing
  rather than logging a warning. Report it as a layer, not a guarantee: it is
  another model and can be fooled.

## Slack and other chat platforms

- Is the text escaped (`&`, `<`, `>`) before posting? Unescaped output can
  contain `<!channel>`, `<!here>`, user mentions, and links whose text shows a
  different address from the one they point to (LLM10).
- Calls to `chat_postMessage` that post model output should set
  `unfurl_links=False` and `unfurl_media=False` explicitly. Unfurling makes
  Slack fetch the URLs to build a preview, which sends any data in the URL to
  its owner without anyone clicking (LLM02).
- `link_names` should not be set.
- Links in the output should be restricted to an allow-list of domains, or
  built by code.
- Apply the same checks to Block Kit blocks and attachments built from model
  output.

## Other sinks

- SQL, shell commands, file paths, `eval`/`exec` and deserialisation built from
  model output: report as injection (LLM10), as for any untrusted input.
- Output passed to another model or agent: record it as an input to that model
  and check its map too.

## Audience

Who will see the output? Compare with who may see each input (LLM02). For
Slack, check whether the code looks at the channel before posting sensitive
results (`conversations.info`, `is_ext_shared`), and whether results only the
requester may see are sent with `chat.postEphemeral` or a direct message.