Benchmark scores reward completion; production systems must reward containment

Benchmark scores reward completion because completion is easy to count. Production systems must reward containment because the cost of an agent is determined not only by whether it reaches the goal, but by what it touches, spends, exposes, and leaves behind on the way. I do not want an agent that completes ninety tasks and turns the other ten into incidents. I want one whose failures stop inside declared boundaries, preserve recoverable state, and arrive with enough evidence for a human or supervisor to take over.

Task success rate is a useful outcome measure. A booking either exists in the right terminal state or it does not; a software patch either passes the tests or it does not. The trouble begins when that binary result becomes the whole reward. An agent can send the requested email after reading unrelated mail, fix a test by weakening the assertion, or finish a data task while making unnecessary destructive calls. The terminal state says “success.” The production state says the system accepted an unsafe path.

This is a construct-validity problem before it is a scoring problem. If a benchmark claims to measure production readiness but defines readiness as completion, it has quietly removed authority, side effects, reversibility, and failure severity from the construct. The resulting leaderboard is precise about the wrong thing. It also directs engineering effort toward increasingly aggressive agents—more retries, more tools, more autonomy—because caution that refuses an unsafe action scores as failure while reckless completion scores as progress.

I would make containment a first-class part of the result rather than a safety footnote. Every run should return at least four dimensions: goal achievement, boundary adherence, recovery quality, and resource boundedness. Boundary adherence asks whether the agent stayed within permitted tools, data, recipients, and state transitions. Recovery quality asks whether it detected inability, stopped cleanly, preserved valid work, and escalated with context. Resource boundedness asks whether steps, latency, tokens, and external calls stayed inside the task budget. A completed task with an undeclared side effect should not receive the same score as a clean completion; a refused task that correctly detects missing authority should not be scored as an ordinary miss.

That requires evaluating trajectories as well as outcomes. The final state answers whether the agent worked. The trajectory reveals whether it looped, retried a non-idempotent action, crossed a trust boundary, ignored a failed verification, or depended on a lucky tool response. I would encode forbidden transitions and protected state directly into the evaluation environment, then inspect both intermediate mutations and the terminal state. Containment becomes executable: no unrelated records changed, no credential scope widened, no external recipient appeared, no retry budget was exceeded, and every irreversible action passed its approval gate.

Repeated trials should expose the same distinction. A metric that asks whether at least one of several attempts succeeds measures capability coverage; it rewards persistence. Production reliability asks whether all repeated attempts remain acceptable. The losing runs matter most here. I want their blast-radius distribution, silent-failure rate, recovery time, and residual state—not just the best sample. One catastrophic trajectory should remain visible instead of disappearing inside a strong average.

The production loop should then optimize the same contract. Canary cohorts limit exposure, semantic gates trigger rollback, and every escaped failure becomes a permanent regression case tagged by severity and containment breach. This aligns benchmark incentives with operating incentives: improve completion, but never by buying it with broader authority or harder recovery.

There is one precise boundary to this claim: for a read-only, low-stakes assistant whose outputs are always reviewed before any external action, completion quality can remain the dominant metric, because its practical blast radius is already capped by the human approval boundary. Even there, cost and silent failure deserve measurement; the full containment apparatus becomes necessary when the system can mutate consequential state.

I would therefore read any agent score as incomplete until it answers two questions together: did the task finish, and did failure remain cheaper than success was valuable? Completion proves capability. Containment proves that the capability belongs in production.