Durable agent execution with checkpoints and cancellation

A reference build that turns agent execution from a fragile script into a durable, resumable, and cancellable workflow. Checkpoints capture intent, tool effects are idempotent, and cancellation is explicit — producing replayable failure evidence instead of silent corruption.

The problem

Most agent execution is a long-running script with no notion of "mid-flight." If the process dies, the network blips, or a tool hangs, the run either restarts from the beginning — repeating side effects — or resumes from an ad hoc checkpoint that may not reflect what actually happened. Cancellation is worse: a SIGINT kills the process, but the tool calls already in flight keep executing, leaving partial state and no record of what was intended.

The failure mode is the same as distributed systems had twenty years ago: no idempotency, no durable log, no explicit cancellation. The agent loop is the transaction coordinator, but it acts like a shell script.

The constraints

This is a reference build, so the constraints mirror production: the run has to survive a restart without duplicating side effects. Cancellation has to propagate to tools that support it and has to be recorded as an event for those that don't. Recovery has to be replay from a durable log, not a hand-rolled "resume from here" marker. And the evidence has to be reconstructible — an auditor should be able to replay the run and see exactly what the agent intended, what succeeded, and where it stopped.

The decisions

Four of them, each a rejection of a fragile default.

Checkpoints are intent, not state. The checkpoint log does not snapshot the agent's memory or context — that is a category error. Instead, it records the intent: the step the agent decided to take, the tool it invoked, the arguments it passed, and the outcome it observed. This is trajectory replay made load-bearing: the log is append-only, time-ordered, and carries enough information to reconstruct what happened without trusting the agent's internal state.

Tool effects are idempotent. Every tool call carries an idempotency key derived from the run id and the step sequence. A database write, an API call, a file operation — each is guarded so that a retry after a crash or timeout does not duplicate the effect. This is the same pattern payment systems use: the key is the proof that "this exact call already happened," and the tool layer enforces it before executing.

Cancellation is explicit, not implicit. A cancellation request does not kill the process — it writes a typed "cancel" event to the log. The agent loop observes the event and stops at the next checkpoint boundary. In-flight tool calls are signalled if they support cancellation; if they don't, the log records that the call was "interrupted mid-flight" so recovery knows not to trust its partial state. This is cancellation as a protocol, not a SIGINT.

Recovery is replay, not resume. After a crash or cancellation, the agent does not "pick up where it left off" from an ad hoc marker. It replays the checkpoint log from the last committed checkpoint, re-observing the same decisions and tool calls, using idempotency keys to skip effects that already succeeded. This is the same pattern workflow engines use: the log is the source of truth, and the agent is a deterministic replay of that log.

The outcome

The build answers the three questions a durable execution system has to answer and a script cannot: what did the agent intend to do, what actually happened, and where did it stop. The checkpoint log is the evidence — append-only, time-ordered, and replayable. Idempotency keys prevent duplicate side effects on retry. Cancellation is a first-class event, not a crash. And recovery is replay from the log, not a hand-rolled "resume from here" that may not reflect reality.

The honest caveat: this is a reference build, not a deployed workflow engine with production traffic. There are no throughput numbers or latency percentiles to show, because the point of the build is the architecture — that durable execution is a discipline, not a library — not a benchmark. A benchmark would tell you it's fast; the pattern tells you whether it can be trusted when things fail.

Reference implementation

The pattern is instantiated as a runnable Python reference implementation — a durable agent loop with a checkpoint log, idempotent tool wrappers, explicit cancellation, and replay recovery. The checkpoint log is append-only (JSONL on disk, or DuckDB for structured queries). Tool wrappers enforce idempotency keys for common operations (HTTP POST, database writes, file operations). Cancellation is a typed event that propagates to the agent loop and to tools that support it. Recovery replays the log from the last committed checkpoint, using idempotency keys to skip effects that already succeeded.

The implementation includes a toy multi-step agent (fetch data, transform, write to database, notify via API) with five self-checking scenarios: normal completion, crash mid-run, timeout on tool call, explicit cancellation, and replay after recovery. A demo script drives the agent through each scenario and prints the checkpoint log, the tool effects, and the recovery trajectory.

The full source — durable loop, tool wrappers, cancellation handler, replay engine, and five scenario scripts — is public at github.com/Dhristhi/agent-durable-execution, under the Apache-2.0 license.

What you can run

git clone https://github.com/Dhristhi/agent-durable-execution
cd agent-durable-execution
uv sync --extra dev
uv run pytest -q
uv run python -m durable_agent.demo
for s in scripts/scenario_*.py; do uv run python "$s"; done