Idempotency is the cheapest reliability you will ever buy

Every distributed system has exactly one universal response to failure: try again. Timeouts, dropped connections, crashed workers, ambiguous responses — the recovery move is always a retry. And a retry without idempotency doesn't repair the failure; it converts it into a duplicate. Two charges. Two emails. Two orders. Of all the reliability mechanisms you can buy — replication, consensus, failover — designing operations to be safely repeatable is by far the cheapest, and it's the one teams most often skip.

The logic is forced on you by delivery semantics. Any platform that guarantees your operation runs — Temporal's at-least-once activity execution is the canonical example — must be allowed to run it more than once, because "did the last attempt complete before the crash?" is often unanswerable. The failure that stings most is the ambiguous one: the request succeeded but the acknowledgment was lost, so the caller retries an operation that already happened. At-least-once delivery and idempotency are two halves of one contract. Accept the first without implementing the second and you've built a duplicate generator with excellent uptime.

The toolkit is old and small

None of this requires invention. Idempotency keys: derive a stable identifier — in Temporal's model, workflow run ID plus activity ID, unchanged across every retry of the same logical step — and let the downstream service deduplicate. Upsert semantics: write operations phrased as "ensure this state exists" rather than "append this effect," so re-execution converges instead of accumulating. And where an API offers no key support, idempotency by validation: a create-then-verify loop that checks whether the effect already exists before re-attempting. When an operation truly cannot be made repeatable, the sagas literature supplies the vocabulary — classify each step as retryable, compensable, or a pivot (the point of no return), run the pivot exactly once with retries disabled, and undo compensable steps with compensation transactions when the sequence fails downstream. That classification exercise — the RIG discipline of sorting effects into reversible, irreversible, and guaranteed — costs a whiteboard hour and saves an incident review.

Agentic systems make this decades-old discipline urgent again, for two compounding reasons. First, agents retry at more layers than traditional services: the platform retries failed activities, the framework retries failed validations — the structured-output literature's retry-with-schema-feedback pattern, where one reask resolves roughly 70% of semantic failures and two resolve about 90% — and the model itself will cheerfully re-issue a tool call it believes didn't land. Second, agent tool calls are side effects chosen at runtime by a stochastic planner. A hand-written service duplicates the one operation its author forgot to protect; an agent can duplicate anything in its tool catalogue. If a tool wired into an agent isn't safe to call twice with the same arguments, it isn't safe to give to an agent at all. That makes idempotency a review gate for tool design, the same way checkpointing is a gate for agent runtime design — the checkpoint gets you back to the failure point; idempotency makes resuming from it harmless.

The honest limit: idempotency isn't free everywhere. Some operations are irreversible and non-repeatable by nature — sending an email, firing a physical actuation — and for those the right pattern is at-most-once execution (a single attempt, no retries) plus explicit failure handling, accepting that you'll sometimes fail without acting rather than act twice. Deduplication state also has real costs: keys need storage, TTLs, and a uniqueness scope, and in high-throughput paths that bookkeeping is measurable. But compare the price lists. Consensus protocols, distributed locks, exactly-once middleware — each costs latency, operational complexity, or both, forever. An idempotency key costs one column and one design conversation. It is the rare reliability property you mostly get by deciding, early, that you want it.