06 · Trust, consent and evaluation · Generative UI Field Guide

Why generated UI needs its own trust model

A hand-built screen is reviewed by a person before anyone uses it. A generated one, at runtime, usually isn't. That changes three things.

The author is a model trained on the web — including every dark pattern on it. Left alone, it reproduces them.
The author can be steered by strangers. Anything in the model's context — a web page, an email, a tool result — can contain instructions. If the interface is generated from that context, an attacker can shape what the person sees.
Looks and behaviour come apart. A generated page can render a perfect confirmation dialog and fire the request before it appears. Appearance-based review doesn't catch it.

The defences stack by rung. Rung 3 (pre-built components) inherits most of its safety from your components. Rung 4 (declarative specs) is bounded by the catalog. Rung 5 (open-ended code) is bounded by the sandbox. Everything, at every rung, is bounded by consent before consequential effects — the one defence that doesn't depend on the generation technique.

Threat 1: injection through the interface

Indirect prompt injection is the defining attack of agent systems. The agent reads untrusted content — a web page, a document, a tool result — that contains instructions, and follows them. Security researchers demonstrated it repeatedly against AI browsers in 2025, including instructions hidden as faint text in images that the agent's OCR path dutifully read.

For interfaces the risk compounds. Injected text can make a generated UI show something misleading ("Your account is locked — verify here") or propose a consequential action that looks like the person's own idea.

Detection doesn't fix this; filters are always one phrasing behind. Provenance does:

Taint untrusted context. Mark everything that came from somewhere other than the user or your own system. WebMCP's draft even includes a hint for flagging untrusted content.
Tainted proposals can't authorise effects. If a consequential action was proposed on the basis of tainted context, it requires a human confirmation rendered by trusted chrome — that's the step where a person re-grounds their intent.
Separate browsing from acting. Start consequential actions only when the person explicitly invokes them, not because a page suggested it.

Threat 2: code that does too much

At rung 5 the sandbox is the security model. MCP Apps, the official standard for UI inside AI hosts, is a good baseline to study:

the view runs in an iframe on a dedicated, different origin that the host controls (Claude, for instance, serves views from per-app hashed subdomains), so it can't reach the host's DOM, cookies or storage;
the spec's iframe uses sandbox="allow-scripts allow-same-origin" — safe only because the origin is different. If you ever render generated HTML from your own origin (a srcdoc frame, say), never combine those two flags: together they let the frame remove its own sandbox. Use allow-scripts alone;
the view talks to the host only through a narrow postMessage channel carrying JSON-RPC;
network access is denied by default: the CSP starts from default-src 'none', servers declare the domains they need, and the host enforces the list;
every effect goes through a host-mediated tool call, so the host can apply consent rules — but note the spec only says hosts may ask for consent on UI-initiated calls. Consent is your job.

What remains is phishing inside the frame: a generated UI that draws a fake login form or a fake "Approve" button. The sandbox can't stop that, because drawing is exactly what the sandbox allows. The answer is trusted chrome: confirmations for consequential effects — the amount, the recipient, the Approve button itself — are rendered by the host, outside the frame, where generated code can't counterfeit them.

Threat 3: a vocabulary that allows harm

At rungs 3 and 4 the catalog is the policy. Whatever isn't in it can't be rendered.

Pin the catalog per surface. A2UI does this explicitly: a surface declares its catalog when it's created, and clients advertise which catalogs they support.
Generate the prompt from the catalog. json-render's catalog pairs each component's schema with a description and generates the system prompt from it, so generation and validation share one source of truth and can't drift.
Report validation errors with a path. "Component CreditCardForm not in catalog at /children/2" lets the agent repair precisely, or lets you fall back to a lower rung visibly.
Beware escape hatches. An Html or Markdown-with-HTML component in the catalog silently reintroduces rung 5.

Lab 03 shows a renderer refusing a component that isn't in the catalog.

Threat 4: manipulation by default

This isn't hypothetical. A CHI 2026 study generated about 1,300 web components with language models and found that a majority contained at least one deceptive design pattern, with prompts emphasising business goals making it worse. Models learned interface design from the web, and the web is full of pre-ticked boxes, confirm-shaming and hidden costs.

Three mitigations, in order of strength:

Catalogs without manipulative components. No pre-checked consent, no fake countdown timers, no "No thanks, I hate saving money" buttons.
Lint rules for pre-selected paid options, asymmetric styling on decline buttons, and shaming copy.
Behavioural checks on what a flow does by default — what's submitted if the person just clicks the primary button through.

Your catalog and your linter are, quite literally, your ethics layer.

Confirmation dialogs cost attention. Spend them everywhere and people rubber-stamp; spend them nowhere and people get hurt. The principle: consent proportional to how hard the effect is to undo. We use a four-rung reversibility ladder:

Class	Example	What the interface owes the person
Reversible by the person	rename a draft, reorder a list	do it, and offer undo
Reversible by an operator	archive a project, cancel an order before shipping	a clear inline confirmation
Irreversible, bounded	send one invoice, charge $20	explicit confirmation showing the exact amount and recipient
Irreversible, unbounded	delete an account, email 40,000 customers, wire money	explicit confirmation plus approval from a human outside the run

Four details turn the policy into a guarantee:

Classify the call, not the tool. sendEmail to one colleague and to a mailing list are different classes. Compute the class from the arguments.
Confirmation before effect. The confirmation must be visible before the request leaves the browser. Present in the DOM afterwards doesn't count.
Approval binding. Approval authorises these exact arguments. If they change between render and execution, the approval is void. Some SDKs now sign approvals over the tool name, call ID and input.
Prefer undo to confirmation where you can. A five-second undo window on a reversible action respects attention better than a dialog.

Lab 05 walks a tool call through the lifecycle and shows what happens when the confirmation comes after the effect.

Accessibility of generated and streamed UI

Streaming is hostile to assistive technology by default: a live region that announces every token is unusable; one that announces nothing hides the answer. Working practice:

mark the streaming container aria-busy="true" and announce once, politely, when it settles;
never move focus into streamed content;
announce state changes ("approval needed", "three files changed"), not the content itself;
give provisional content an accessible name that says it's provisional.

Rung 3 and 4 output inherits whatever accessibility your components have — one more argument for them. Rung 5 output must be audited in a real browser, because the model chose the semantics. And remember Module 05: accessibility failures are agent-operability failures too.

Brand and design-system conformance

"Your brand is a rule, not a vibe." Enforce it structurally:

props accept tokens, not raw values: tone: "danger", not color: "#f00";
catalogs expose layout primitives, not CSS;
rung-5 output is constrained to a provided stylesheet or linted for off-palette colours and off-scale spacing.

Evaluating what generated UI does

Here is an evaluation stack, cheapest first. Each layer catches failures the previous one can't.

Schema and catalog validation — every spec, every time. Catches shape errors.
Render tests — specs to DOM snapshots and accessibility trees. Catches broken rendering and missing semantics.
Stream fuzzing — truncate and reorder streams at every boundary. Catches flicker, premature actions, partial-JSON crashes.
Behavioural checks in a real browser — drive the interface, pause each outgoing request, and ask: what was visible at that instant, what's this effect's class, does the payload match what was shown? Catches the dialog-after-delete, the $40-shown-$4,000-sent, and the default that submits more than it shows.
Human preference evals — experts compare outputs. Catches "technically correct, actually bad".

Two techniques make the stack practical. Deterministic replay: record the event stream once (AG-UI events, A2UI messages) and replay it forever as a regression test — the projection should come out identical. Effect grading: judge each network effect against its reversibility class and the visible state at dispatch time, rather than judging pixels.

Layer 4 is what RUI does: it opens the page in Chrome, pauses requests at the network layer, reads the page while the request is held, and decides PASS, BLOCK or REVIEW with the observed evidence behind every reason. Missing evidence routes to review, never to a quiet pass.

A pre-launch checklist

Every generative surface has a declared rung, and a reason it isn't one lower.
The catalog is pinned, versioned, and contains no escape hatch to raw HTML.
Rung-5 output runs on a dedicated origin (or a same-origin frame with allow-scripts only), with egress denied by default.
Every consequential confirmation is rendered by trusted chrome.
Every action has a reversibility class computed from its arguments.
Confirmations are visible before effects, and approvals bind to exact arguments.
Untrusted context is tainted and can't authorise effects on its own.
No secrets in the model's context.
Streamed content is announced on settle and never steals focus.
A recorded stream replays to the same projection in CI.
Behavioural checks run in a real browser before release.

Check yourself

Your app renders rung-5 charts in a srcdoc iframe — same origin as your app — with allow-scripts allow-same-origin. What can the generated code now do, and why is the same flag pair acceptable in MCP Apps?
A generated UI shows "$40" and submits "$4,000". Which layer of the evaluation stack catches it, and which layers don't?
Classify: archiving a Slack channel; sending a calendar invite to 3 people; sending it to 3,000; deleting a draft with undo.
Why does detection-based injection filtering fail, and what replaces it?
A teammate proposes adding an Html component "just for edge cases". What do you say?