Interfaces agents can use

The other direction

Modules 01–04 are about agents producing interfaces. This one is about agents consuming them. When a browser agent books a table, a coding agent clicks through your admin panel, or an assistant inside ChatGPT operates the embedded app you shipped, the interface is no longer read only by a person. It has a second audience that perceives it through a very different, very lossy channel.

The two directions are converging. The interface an agent generates for you today is the interface another agent will try to operate tomorrow. So the question isn't "should we design for agents?" — they're already here — but "through which channel will they perceive us, and what will they get wrong?"

Three ways to perceive a page

Agents perceive interfaces through one of three families of channel. Each trades fidelity, cost and risk differently.

Channel	What the agent gets	Fidelity	Cost	Example
Pixels	screenshots; it guesses layout and reads text	medium — OCR and inference	high — image tokens, many steps	computer-use models, screenshot-driven browser agents
Structure	the DOM or the accessibility tree: roles, names, states	high if the page is accessible, near zero if not	medium	most browser agents; Playwright-style automation
Contracts	declared, typed capabilities it can call directly	contract-level	low	WebMCP tools, MCP Apps, data-tool affordances

Pixels are universal and the worst of the three. A screenshot agent clicking "Delete" has to infer that the red button deletes, infer what it deletes, and hope the layout didn't shift between screenshot and click. It's how a person with no context would use a page through a keyhole.

Structure is where most agents live today, which makes the accessibility tree the de facto API of the web for agents. And contracts are where things are heading: the page says, in a machine-readable way, "here are the things you can do, here are their arguments, here is which ones change data".

Accessibility debt is agent debt

Here is the most practical idea in this module. An element that a screen reader can't understand, an agent can't understand either — because both are reading the same accessibility tree.

Consider two versions of the same button:

<!-- invisible to both audiences: role "generic", no name -->
<div class="btn-red" onclick="del()"><svg>…</svg></div>

<!-- legible to both: role "button", name "Delete account", described consequence -->
<button aria-describedby="del-note">Delete account</button>
<p id="del-note">Permanently deletes your data. This can't be undone.</p>

To the accessibility tree, the first is generic with no name. A screen-reader user hears nothing useful. An agent sees an unlabelled, unclickable-looking node and either misses it or guesses. The second is button "Delete account" with a description that states its consequence — precisely what both audiences need.

So the first and cheapest step in making an interface agent-ready is the one you should have done anyway: semantic HTML, real names on controls, visible labels on inputs, states exposed (aria-expanded, aria-checked, disabled), and no meaning carried only by colour or position. Lab 08 shows the same page in human view, accessibility-tree view and tools view, with a toggle that breaks it the way real pages break.

Declaring tools: WebMCP

Structure tells an agent what's on the page. It doesn't tell it what the page can do in a reliable, typed way. That's the job of tool declarations.

WebMCP is a draft web API, incubated in a W3C community group, that lets a page register tools for agents — essentially MCP, but exposed by the page in the browser. In the current draft it hangs off document.modelContext (earlier drafts and builds used navigator.modelContext, so feature-detect both):

const mc = (document as any).modelContext ?? (navigator as any).modelContext;
mc?.registerTool(
  {
    name: "invoices.sort",
    title: "Sort invoices",
    description: "Sort the visible invoice table",
    inputSchema: {
      type: "object",
      properties: { column: { enum: ["date", "amount", "client"] }, dir: { enum: ["asc", "desc"] } },
      required: ["column"],
    },
    annotations: { readOnlyHint: true }, // changes the view, not the data
    execute: async ({ column, dir = "asc" }) => { setSort({ column, dir }); return `sorted by ${column}`; },
  },
  { signal: controller.signal } // abort to unregister
);

Three details matter more than the syntax:

1. Annotations carry consequence. A hint that a tool is read-only, or destructive, lets an agent (and its host) decide how much confirmation to require. That's the reversibility idea from Module 06, declared at the source.
2. Registration is scoped to the UI's lifetime. The tool exists while the component that owns it is mounted. When the table disappears, so does invoices.sort. Tools stay honest about what the page can do right now.
3. It's a draft. Names and details have changed across drafts and may change again. Build against it behind feature detection, and keep the accessibility tree solid as the fallback.

MCP Apps and the OpenAI Apps SDK have their own versions of the same idea for UI embedded in AI hosts: an app can update what the model knows about it, and can call tools the model can't see. Same principle — explicit, typed, scoped.

Dual-audience components

Put the pieces together and you get the design pattern this lab is built around: one source of truth, two projections. A sortable table renders clickable headers for the person and registers a typed sort(column, direction) tool for the agent. Both call the same setSort. Neither path can drift from the other, because there's only one implementation.

function InvoiceTable({ rows, sort, setSort }) {
  useAgentTool({ name: "invoices.sort", readOnly: true, schema: SortSchema, run: setSort });
  return (
    <table aria-label="Invoices" data-ui="invoice-table" data-tool="invoices.sort(column, dir?)">
      <thead>
        <tr>{COLUMNS.map((c) => <th key={c}><button onClick={() => setSort({ column: c })}>{c}</button></th>)}</tr>
      </thead>
      {/* … */}
    </table>
  );
}

Agentable UI packages this as a convention: every component carries a stable data-ui name, and anything an agent can call carries a data-tool signature, with risky actions requiring confirmation. The broader principle doesn't depend on any library: give every capability a name, a type and a declared consequence, and expose it to both audiences from one implementation.

Keeping the reverse channel honest

The reverse channel turns what's on screen into what the model believes. It's lossy, it goes stale, and it's an attack surface. Three rules:

1. Curate, don't dump. Sending the whole DOM to a model burns tokens and leaks whatever happens to be on the page. Export an agent view instead: a deterministic serialisation of the interactive surface — role, name, state, stable ID, available tools. It's cheaper, more reliable, and doubles as an accessibility audit.

2. Version the context. Stamp everything you send with the version of the screen it describes. If a person selects rows 3 and 7, says "refund these", then deselects row 7, the agent's proposal refund([3, 7], basedOn: v41) should be caught and re-confirmed against v42. This is the guarded patch from Module 03, applied to context.

3. Assume anything in context can leak. Widget state, readable app state, tool output — if it's in the model's context, a prompt injection elsewhere in that context can ask the model to repeat it. Keep secrets, tokens and payment details out of the context plane entirely.

And treat selection as pointing. When someone says "make this bigger", the selection is the referent. Send it explicitly; don't make the model reconstruct it from a screenshot.

Designing for operators that aren't people

A few habits make interfaces dramatically easier for agents to operate safely — and every one of them also helps people:

• Explicit actions over gestures. Drag-to-reorder needs a "Move up / Move down" equivalent. Swipe-to-delete needs a button.
• Stable names. Agents (and tests) rely on names that don't change between releases. Don't rename "Archive" to "Stash" for flavour.
• Declared consequences. Say what a destructive action does, in text, next to the control. Better still, declare it in tool metadata.
• Idempotent operations. Agents retry. "Pay invoice #1042" should be safe to call twice; "Pay next invoice" is not.
• Consent that requires a human. For irreversible effects, the confirmation must be something an agent can't satisfy by clicking — host-rendered, outside the agent's reach, or bound to a person. Otherwise "confirm" is just one more button for the agent to press.

Check yourself

1. Rank pixels, structure and contracts for an agent that must cancel a subscription. What breaks in each?
2. An icon-only "trash" button has an onClick and no label. Describe what a screen reader and an agent each perceive.
3. Why should a WebMCP tool be unregistered when its component unmounts?
4. Sketch a dual-audience date picker. What's the single source of truth, and what does each audience get?
5. A support agent's context includes the full customer record "for convenience". What's the risk, and what's the fix?
6. Why is "the agent clicked Confirm" not consent for an irreversible action?

Go deeper

• Lab: 08 · the page as an agent reads it
• Patterns: R1 Readable-State Contract, R2 Selection as Deixis, R3 Dual-Audience Component, R5 Agent View Projection
• Specs and tools: WebMCP draft, MCP Apps, Agentable UI, WAI-ARIA Authoring Practices
• Next: 06 · Trust, consent and evaluation