Agent testing is simulation engineering — record-replay beats mocks

Agent testing is simulation engineering. A hand-written mock tells you whether code follows the branch you predicted; a useful agent test asks whether the system still reaches the right state when language, tools, and prior actions interact in ways you did not script. For multi-step agents, record-replay beats invented mocks because it preserves the awkward details of reality — real arguments, real response shapes, real ordering, and the exact point where a plausible trajectory went wrong.

The mistake is treating the model response as the unit under test. An agent run is a trajectory through a changing world: context produces a decision, the decision invokes a tool, the tool mutates state, and that state conditions the next decision. A mock that always returns the same flight options or account record regardless of the arguments can let a broken agent pass. It tests control flow against a stage set. Simulation engineering instead asks what fidelity the world model needs for this failure mode.

Record reality, then make it replayable

I would start with production run fixtures, not developer imagination. Capture the complete interaction: conversation history, tool calls and arguments, tool results, final output, plus the model, prompt, and tool versions that produced it. Select fixtures deliberately — negative-feedback runs, high-risk tool invocations, latency outliers, and a representative sample — then replay them against the candidate system. The output is not a single pass/fail score but a structured behavioural diff: which tool changed, which argument drifted, which call disappeared, and where the first divergence occurred.

That comparison must be strict in the right places and tolerant in the right ones. Tool names, consequential arguments, state mutations, and call ordering deserve exact assertions. Natural-language responses need semantic comparison because harmless rephrasing is not a regression. Most importantly, score the world after the run. If an agent says it issued a refund but the simulated ledger and order record did not change, eloquence should score zero. Final-state and intermediate state-transition assertions catch the class of agents that narrate success without producing it.

Replay also needs isolation. Recorded tool responses should be keyed by normalized calls, live network access should be blocked, and an unseen call should fail loudly instead of escaping to production. Version the fixtures with the agent and keep the baseline attributable to a dataset, prompt, model, and tool set. You then have something more useful than a mock suite: an executable memory of approved behaviour, built from cases the system actually encountered.

Where mocks still win

Here is the precise concession: hand-authored mocks are better for narrow unit tests and deliberate fault injection. If I am checking JSON parsing, retry logic, a 500 response, a timeout, or the structure of an SDK invocation, a fixed response is cheaper, clearer, and more diagnostic than a recorded trajectory. Record-replay should not replace the bottom of the testing pyramid. It should replace the fiction that bottom-layer mocks provide system-level confidence. The other concession is that recordings decay: schemas, policies, prices, and tool behaviour change. A deterministic replay against stale fixtures is perfectly reproducible evidence about a world that no longer exists, so fixtures need freshness checks, live re-recording, and explicit retirement.

For paths that no recording covers, build a stateful simulator rather than a larger pile of static responses. Let simulated tools share state, validate outputs against schemas, inject failures, and expose checkpoints for assertions. Run stochastic cases more than once and treat variation statistically rather than demanding byte-identical trajectories from a probabilistic model. This is why I call the discipline simulation engineering: the hard question is not "what should this mock return?" It is "which parts of reality must this test preserve, which may vary, and what observable state proves the task was actually completed?"