Computer-use agents are the integration of last resort — design for when APIs don't exist
A computer-use agent — screenshot in, click and keystroke out — is the most expensive and most fragile way one piece of software can talk to another. That doesn't make it useless; it makes it the integration of last resort, and the design stance follows directly: reach for pixels only when no API exists, and when you must, engineer the fallback like the hazard it is.
The formalism explains the fragility. A computer-use agent is solving a partially observable decision problem: the screenshot is an observation, never the state, and the true application state is permanently hidden behind it. Everything an API integration gets for free — typed inputs, structured errors, an explicit contract — the GUI agent must infer from pixels on every step of a perception-reasoning-action loop. Even the observation itself is a compromise you choose: raw screenshots for fidelity at heavy token cost, accessibility trees for structure at the price of losing visual layout, or annotated hybrids that bridge the two. The characteristic failure modes fall out of that inference: GUI hallucination, where the agent confidently references an element that isn't on screen, and loop traps, where it repeats an action sequence because nothing in the observation tells it the action didn't work. Serious harnesses ship loop detection and screenshot-history truncation as standing components — machinery an API caller simply doesn't need.
The economics say the same thing louder. Task costs across the current systems range from about two and a half cents per task for a small specialist model to over ninety cents for a frontier one — one to two orders of magnitude above the equivalent API calls, with worse tail reliability. And the security posture is the worst of any agent modality: a computer-use agent reads whatever the page shows it, so adversarial instructions embedded in web content hijack goals directly, which is why critical-point detection — pausing for human confirmation before irreversible actions like payments, sends, and deletes — plus per-task session isolation are table stakes, not enhancements.
The last resort is real work, done honestly
None of this argues that computer-use doesn't matter. The long tail of enterprise software — legacy Windows apps, government portals, vendor systems with no integration surface — is exactly where agents create value, and it has no APIs to offer. That's why the serious engineering there looks the way it does: dual-agent architectures that separate global planning from low-level UI execution, and executors that prefer OS accessibility APIs over raw coordinates when the platform offers them — each a way of clawing back structure from an unstructured interface. Even the training story is a last-resort story: there are no large corpora of human GUI demonstrations, so synthetic trajectory pipelines generate and verify their own.
The concession: capability is improving fast enough that the reliability half of my argument is decaying. The strongest current models now clear human baselines on desktop-automation benchmarks, and cost curves for small specialist models keep falling — for some domains, one general screen-level agent may soon beat maintaining dozens of bespoke API integrations. But notice where the ecosystem itself is heading: proposals for websites to declare their agent-accessible actions, cryptographic identity for agent traffic — the web growing an API layer precisely because everyone involved understands that pixels are the interface you use when you have no better offer. Build the escape hatch; label it as one; and when you control both sides of the interface, give your agent a real contract instead.