Browser agents should prefer accessibility trees before pixels

Browser agents should prefer accessibility trees before pixels. I want the agent to begin with the interface’s declared semantics—roles, names, states, and interactable elements—and escalate to visual reasoning only when those semantics are absent or misleading. Starting from screenshots turns every page into an image-recognition problem before the system has proved that an image is necessary.

The structured path is economical because it narrows the action space. A browser runtime can filter the DOM to visible, interactable elements and serialize a button as a compact indexed object rather than send the whole page. The source material puts these representations in the range of hundreds to a few thousand tokens. That gives the model a bounded vocabulary: click item 12, type into item 7, select an option. It also makes the resulting action easier to validate than an unqualified coordinate.

Accessibility structure is not merely a token-saving trick. It carries distinctions pixels force the model to infer—whether a control is a button, whether a field has a name, whether an item is disabled, and which text labels which action. Systems such as Stagehand use accessibility and DOM context to generate selectors, then verify uniqueness and retry when the selector matches nothing or too much. That is closer to an integration contract than to cursor imitation.

Pixels still matter because web interfaces routinely under-specify themselves. Canvas applications, visual editors, charts, maps, custom controls, and badly implemented single-page applications may expose little useful structure. Screenshots reveal spatial grouping and visible state that a filtered tree can miss. Set-of-Marks techniques bridge the two worlds by placing numbered overlays on visible elements, letting the model reason visually while selecting from a structured index.

This is why I prefer an escalation ladder rather than a doctrinal choice. First inspect the accessibility tree and structured DOM. Then add a screenshot when visual arrangement changes the decision. Use coordinates only for the residual surface that cannot be addressed semantically. The hybrid evidence is stronger than either source alone—the tree proposes what can be acted on, while the image checks whether the proposal corresponds to what a user can actually see.

The hierarchy also improves recovery. A selector tied to a named control can be regenerated when the page changes; a coordinate tied to yesterday’s layout cannot explain what it intended. DOM hashes and instruction-to-selector caches can make stable steps deterministic and cheap, while changed structures trigger fresh interpretation. That makes browser automation an instance of API design under uncertainty, not a sequence of fragile mouse recordings.

Evaluation should follow the same principle. I do not consider a plausible click trace proof of success. The runtime should inspect authoritative resulting state, preserve session replay, and distinguish “the intended control was found” from “the effect was committed.” This is the same argument behind verifying state rather than clicks: observation and action are means; the postcondition is the contract.

There is one bounded concession: when the task is intrinsically visual, or the application exposes no trustworthy semantic structure, pixel-first reasoning can be the correct starting point. A drawing canvas or remote desktop may leave nothing useful to traverse. The concession does not extend to ordinary forms and navigation where accessible structure already names the controls more precisely than vision can.

The production default should therefore be semantics first, hybrid when needed, and coordinates last. Browser control is already the integration of last resort; choosing the least structured observation method inside that fallback compounds its cost and brittleness. A browser agent becomes easier to govern when every escalation can answer why the structured interface was insufficient.