Agent QA should test recovery paths more than happy-path completion

I think agent QA should test recovery paths more than happy-path completion. The happy path proves that one trajectory can work under cooperative conditions. Production confidence depends on what happens when a tool times out, a schema changes, an instruction is malicious, state becomes stale, or a run is interrupted after an external effect. An agent that finishes ordinary tasks but cannot stop safely or resume honestly is not high quality—it is merely fortunate.

This changes the unit of testing. I do not want only prompt-and-answer cases; I want a state machine whose rules represent user messages, tool calls, file mutations, retries, checkpoints, and injected failures. Invariants should be checked after every transition: the process remains live, paths cannot escape the workspace, tool calls stay bounded, credentials do not appear in output, and committed effects are not duplicated. The result is closer to simulation engineering than to a collection of golden responses.

Property-based testing is valuable here because failure paths compose combinatorially. Hypothesis and fast-check can generate unusual values and sequences that an example suite will not imagine, then shrink a forty-step failure to the three transitions that actually matter. That shrinking is operationally important. A minimal counterexample turns “the agent sometimes loses state” into a reproducible sequence such as checkpoint, timeout, retry. The saved failure then becomes a permanent regression fixture rather than an anecdote in an incident review.

The properties need judgment. Automatically generated properties are often trivially true, and code coverage says nothing about whether the suite distinguishes correct recovery from plausible failure. I would combine example tests with property tests and mutation testing: alter a permission check, suppress a checkpoint write, invert a retry classification, or remove an escalation branch, then ask whether the suite notices. A surviving mutant is evidence that the recovery contract has not actually been specified.

Agentic QA can help explore the application, generate candidate tests, correlate traces with code changes, and maintain brittle UI paths. But I would not let the testing agent silently “heal” a test until the changed locator or expectation has been reviewed against product intent. Self-healing that makes a suite green by accepting altered behavior is automated evidence destruction. Adaptation is useful only when the invariant remains explicit.

The scorecard should follow the same priorities. Defect escape rate matters, but so do explicit-failure rate, successful rollback, preserved-progress rate, duplicate-effect rate, bounded tool-call count, and escalation quality. These measures extend failure-mode design into executable evidence. They also support containment as a production objective: an incomplete run that stops inside its authority can be better than a completed run that crosses it.

There is one precise boundary to this claim: deterministic, read-only, single-step tasks still deserve a larger share of ordinary input-output tests because their recovery surface is small and the human review boundary caps the damage. Even there, malformed inputs and explicit failure remain worth testing. My emphasis applies most strongly once an agent mutates state, waits on humans, chains tools, or survives beyond one request.

I would therefore build the QA plan from failure transitions inward. Name the consequential invariants, generate sequences that threaten them, inject faults at commit boundaries, shrink failures, and retain the counterexamples. Happy-path completion can show that the agent is capable. Recovery-path testing shows whether that capability can be operated.