Recovery plans start with state ownership, not restart commands

Recovery plans start with state ownership, not restart commands. Restarting is safe only when the system knows which state is authoritative, which effects have already committed, and how unfinished work can be reconstructed. Without those answers, a restart is not recovery—it is another write into an uncertain history.

The first question is not “which service is down?” but “what truth does this service own?” A process may own nothing durable and be freely replaceable. It may own an in-memory lease whose expiry coordinates other workers. It may have acknowledged a queue message before persisting an outcome. It may have sent an external payment after recording only an intention. These states demand different responses even if the same container terminated in every case.

I want the recovery map to name state by authority and lifecycle. The database may be authoritative for business records, the event log for ordered facts, object storage for artifacts, and a checkpoint store for resumable execution. Caches and replicas must say what can be discarded and how freshness is re-established. Every boundary needs a commit rule: when does an attempted action become a fact, and which record proves it?

Side effects make ownership unavoidable. If a worker times out after calling an external system, local state alone cannot tell me whether retry is safe. Idempotency keys can make repeated requests converge. Transactional outboxes can couple a local commit to eventual publication. Sagas can attach compensating actions to multi-step work. None provides magic rollback; each makes a different ownership promise about intent, completion, and repair. Idempotency is cheap reliability precisely because it turns ambiguous retries into a designed state transition.

Ordering matters too. A recovered system must distinguish events that may be replayed, effects that must be suppressed, and derived views that can be rebuilt. Checkpoints need a schema version and a known relation to committed effects. Logs need retention and replay ownership, not merely durable bytes. A backup proves that data existed at a point in time; a tested restore proves that the running system can interpret it, reconcile later effects, and return to service.

This is why durable workflow histories are executable recovery contracts. They preserve enough transition history to decide what happens next after interruption. The contract should expose terminal and non-terminal states, retry limits, cancellation, compensation, and operator intervention. Recovery then becomes a continuation chosen from evidence, not a blanket instruction to start over.

My runbook therefore begins with invariants: no effect occurs twice, no acknowledged work disappears, tenant state never crosses boundaries, and an operator can identify the last confirmed transition. It then lists evidence sources, reconstruction steps, degraded modes, and decision owners. Restart commands appear only after the plan states what they preserve, invalidate, or replay. A successful drill checks user-visible outcomes and reconciles records—not merely that pods returned to Ready.

There is one precise concession: restarting is a complete recovery plan for a genuinely stateless, deterministic component whose inputs remain available and whose outputs are either uncommitted or idempotent. The boundary is strict. Hidden local queues, leases, caches that influence decisions, nondeterministic re-execution, or external effects make the component stateful in the sense that recovery cares about.

I judge recovery architecture by whether the team can explain the system after a partial failure. Which facts survived? Which work is safe to repeat? Which outcomes require compensation? Which evidence is missing? Agents need checkpoints, but checkpoints are useful only inside an ownership model that binds them to effects. Processes are disposable. State, authority, and consequences are not.