Agent webhooks are distributed transactions wearing HTTP clothes

Agent webhooks are distributed transactions wearing HTTP clothes. I treat every callback as a cross-system attempt to move one fact — an agent task reached a terminal state — from my authority into somebody else’s. The POST is only the transport. The real problem is preserving that fact across crashes, ambiguous acknowledgements, duplicate delivery, hostile replay, and a receiver I do not control. Once I see the callback this way, webhook design stops being endpoint plumbing and becomes transaction design.

The contract begins before delivery. An agent task may run for seconds, minutes, or wait on external input, so I do not hold the submission connection open and pretend synchronous HTTP can represent its lifetime. I accept the command, persist a task identifier, return 202 Accepted, and let the caller observe progress separately. That 202 is not success; it is a durable promise to execute and eventually report a terminal outcome. Completed, failed, and canceled must all be deliverable facts, not just branches inside a worker.

I separate execution from notification with a transactional outbox. In the same local transaction that records the task’s terminal state, I write a delivery record containing the event ID, destination, payload version, and attempt metadata. A dispatcher reads that outbox later. This closes the classic gap where the worker commits completed and crashes before enqueueing the callback. I cannot make the task database and the customer’s endpoint one ACID transaction, but I can make “terminal state exists” and “delivery is owed” one atomic local decision.

The dispatcher then runs a delivery state machine, not a fire-and-forget loop. A record moves from pending to in-flight, delivered, retry-scheduled, or dead-lettered, with attempt count, next-attempt time, last status, and last error persisted. Timeouts and 5xx responses consume a bounded retry budget with exponential backoff and jitter; most 4xx responses fail fast unless the contract names an exception. A 2xx acknowledgement means the receiver accepted the event, not that every downstream business effect has completed.

Retries make duplicates inevitable, so sender and receiver need a shared idempotency story. I keep one stable event ID across every attempt and send it as the idempotency key; the receiver stores that key within an agreed TTL and returns the prior outcome when it sees the event again. The sender must never mint a fresh identity for a retry. This does not manufacture exactly-once delivery. It gives at-least-once transport and effectively-once handling where the deduplication boundary is explicit — usually the cheapest reliability property in the system.

Security belongs inside the protocol too. I sign the timestamp and raw request body with HMAC-SHA256, include a key identifier for rotation, and require constant-time signature comparison. The receiver rejects timestamps outside a narrow replay window — five minutes is a practical default — then applies the event-ID deduplication check. HMAC proves the payload came from a holder of the shared secret; the timestamp limits captured-request reuse; the idempotency record catches replay inside the permitted window. None substitutes for TLS, and TLS substitutes for none of them.

A dead-letter queue is not a graveyard I occasionally inspect. It marks the boundary where automatic retry has stopped and an operator or compensating workflow must decide what the undelivered fact means. If the callback only projects an already-committed terminal state, replaying delivery is usually safe. If receipt triggers an irreversible effect, such as releasing funds or sending a regulated notice, I classify that step as a pivot and define compensation before launch. Cancellation deserves the same rigor: canceled is a terminal state to publish, while cleanup and reversal remain separately observable actions.

There is one bounded exception to this architecture: for a low-value, internal notification whose source state is queryable and whose loss has no business consequence, polling can be simpler than an outbox-backed webhook. That concession ends when the callback triggers money, entitlement, compliance evidence, or any irreversible action. In those systems, I want the full chain: 202 acceptance, durable task state, atomic outbox, explicit dispatcher transitions, bounded retries, idempotent receipt, signed requests, replay limits, and an owned dead-letter or compensation path. HTTP carries the message; transaction discipline carries the promise.