Recovery architecture begins with an explicit consistency-anomaly budget

Recovery architecture begins with an explicit consistency-anomaly budget. “The system recovers” is not a useful promise until it states which stale reads, duplicates, lost updates, partial effects, and unavailable operations may occur during recovery—and for how long. Without that budget, teams optimize restart time while leaving product correctness undefined.

Consistency models are product decisions because anomalies are user experiences. Recovery makes those experiences concentrated and visible. A failover may expose stale replicas. A replay may duplicate an external effect. A restored snapshot may forget recent writes. A partition may keep reads available while rejecting mutations. These are not equivalent failures, and a single recovery-time objective cannot describe them.

I use “budget” deliberately. An error budget permits a bounded amount of unavailability; an anomaly budget permits bounded inconsistency by class, audience, and duration. An account balance may allow no lost updates and only read-your-writes staleness measured in seconds. An analytics dashboard may tolerate an hour of stale aggregates. A recommendation feed may accept reordering but not cross-tenant leakage. The acceptable set must be named before architecture can enforce it.

The budget then shapes mechanisms. If duplicate effects are forbidden, every operation needs a stable idempotency key and a deduplication horizon longer than the retry horizon. If stale reads are acceptable but lost writes are not, quorum and failover policy can favor durable acknowledgement over immediate replica visibility. If partial completion is permitted, the product needs a visible state and a reconciliation path instead of pretending the transaction was atomic.

Agent systems make the distinction sharper. A run can checkpoint its reasoning and still lose the confirmation that a payment or email completed. Replaying from the checkpoint may repeat the effect; skipping it may leave the task incomplete. Checkpointed execution therefore needs commit states such as proposed, attempted, confirmed, and reconciled. “Resume from last step” is unsafe when the last step crossed a side-effect boundary.

Recovery also has temporal layers. A timeout says when the caller stops waiting, not whether the operation stopped. Exponential backoff and jitter reduce amplification, but retries at multiple layers can still multiply requests. Circuit breakers contain a failing dependency; bulkheads keep its exhaustion from consuming every worker. Load shedding protects recovery capacity. Each mechanism spends a different part of the anomaly budget, so they must be designed as one policy rather than accumulated as libraries.

Testing follows directly from the declared anomalies. Inject coordinator failure after durable acknowledgement but before response. Restore from a snapshot while new writes arrive. Partition a replica, replay a message beyond the deduplication window, and kill a worker between tool execution and checkpoint commit. The assertion is not merely that service returns. It is that observed anomalies remain inside the promised classes and windows, and that reconciliation closes them.

There is one precise concession: not every system needs a formal document with a matrix of isolation phenomena. A stateless, read-only service over disposable data may recover acceptably through retry and cache expiry. The stronger discipline starts wherever recovery can alter money, authority, durable user state, or an autonomous agent’s external effects.

Reliability is failure-mode design, and an anomaly budget turns that inventory into an operational contract. Name what may be wrong, who may observe it, how long it may persist, and what proves reconciliation. Then select snapshots, logs, quorum, retries, and failover around those answers. Recovery is not the return of green dashboards. It is the controlled restoration of trustworthy state.