Computer-use evals should verify resulting state, not clicks

A computer-use eval should verify the state the task was meant to change, not the clicks the agent happened to make. A plausible cursor path is not completion. The agent can press the expected button while the form is rejected, save to the wrong account, duplicate an order, or announce success over an unchanged database. Scoring the interaction instead of the result rewards choreography.

GUI agents operate through partial observations. A screenshot shows rendered pixels, not the application’s authoritative state; an accessibility tree exposes structure, not every business rule; the next screen may reflect a stale cache or optimistic update. That is why action matching is weak evidence. Two valid trajectories can use different controls, keyboard shortcuts, or navigation orders and reach the same correct state. Conversely, a benchmark-perfect sequence can fail because a permission, modal, validation rule, or delayed write changed what the actions accomplished.

The WebArena design points toward the right contract: use locators and evaluators against the final environment, including database state where possible. For a purchase task, verify the order record, item, quantity, account, and absence of duplicates. For an issue update, inspect the repository or application object, not the toast message. For a file operation, verify path, contents, and metadata. Exact-match, must-include, and fuzzy textual checks are useful only when text is genuinely the task output; they should not stand in for functional correctness.

I would separate four layers of evidence. First, outcome: did the intended state exist? Second, invariants: were unrelated records, permissions, balances, and files preserved? Third, process safety: did the agent cross a forbidden boundary, expose sensitive data, or bypass an approval even if the outcome was correct? Fourth, efficiency: how many steps, retries, tokens, and elapsed seconds did the successful run consume? Completion without invariants is unsafe; a clean process without the outcome is incomplete; efficiency matters only after both.

This makes the benchmark infrastructure more demanding. Environments must be resettable, seeded with known identities and permissions, isolated from live systems, and instrumented with verifiers below the GUI. Multi-tab and long-horizon tasks need correlation across pages, while visual tasks need stable semantic targets without overfitting to one layout. The evaluator should also detect impossible tasks rather than rewarding endless attempts, because recognizing that the requested state cannot be reached is a legitimate terminal outcome.

The same principle should govern production tests. Record-replay preserves realistic trajectories, but replay earns confidence only when it asserts shared state after each consequential boundary. Recovery-path testing should interrupt the agent after a click but before confirmation, then ask whether retry duplicates the effect or resumes honestly. Pixels are how the agent acts. State is how the system proves what happened.

There is one precise concession: action-level checks are appropriate for diagnosing perception and grounding. If the research question is whether an agent can locate a control, interpret text embedded in an image, or select the correct menu item, the click itself is the measured capability. That diagnostic should remain a component score. It should not be reported as end-to-end task success when the application state is the user’s actual objective.

Computer use is the integration of last resort because it replaces typed contracts with inference over interfaces. Evaluation should claw the contract back from the other side: define the intended state, inspect it through authoritative channels, and preserve the trajectory only as evidence explaining how the state changed. Evals are the executable definition of working. For GUI agents, working means the world is correct after the cursor stops.