Response APIs make conversation state an infrastructure dependency

Response APIs make conversation state an infrastructure dependency. When a provider stores prior responses, preserves reasoning continuity, hosts tools, and runs work in the background, it is no longer merely converting an input into an output. It has become part of the application’s state, execution, and recovery architecture—and I want that dependency named before convenience turns it invisible.

The shift begins with identifiers. A previous-response identifier can connect one turn to another, while a conversation identifier can provide platform-managed threading. Branches may form a directed graph rather than a single transcript. Those handles now decide which history influences a request. Losing one breaks continuity; attaching the wrong one joins contexts that should remain separate. An apparently small SDK field becomes an ownership boundary.

Server-managed state reduces repeated payloads and can improve cache use because the provider already holds context and reasoning artifacts. It also changes what an application can reconstruct independently. If the durable record is only a chain of remote identifiers, local logs may know that a response occurred without containing enough information to replay, migrate, or explain the effective context. The application still owns the user promise even when the provider owns the continuation mechanism.

Hosted tools deepen the coupling. Web search, file retrieval, code execution, image generation, and remote tool calls can emit polymorphic output items alongside messages and reasoning. A response is therefore an execution record, not just text. The application must preserve which item proposed an effect, which tool committed it, what citations accompanied it, and which parts can be retried. Strict output shape helps integration, but it does not assign lifecycle ownership.

Background execution makes the dependency operational. Long-running responses have statuses, polling, and terminal states; streaming can carry sequence numbers that allow a client to resume delivery. Now provider availability affects not only a request but work already in flight. Cancellation, expiry, partial output, duplicate polling, and callback reconciliation need explicit behaviour. Otherwise a client cannot distinguish “still running” from “state lost” or “result produced but not observed.”

Retention and privacy constraints must shape the design at the same level. The source architecture describes server-managed storage with a bounded retention period and notes that zero-data-retention constraints conflict with background execution. That is not a configuration footnote. It is a product decision about which workflows may use hosted continuity, where regulated data may exist, and whether a task can survive beyond a synchronous request.

I therefore keep an application-owned state envelope around provider state: user and tenant identity, purpose, provider and model, response or conversation identifiers, parent and branch, tool-effect ledger, retention class, and recoverability status. This does not duplicate every token. It preserves enough semantics to enforce conversation ownership, audit effects, and migrate or fail safely when the remote state is unavailable.

There is one bounded concession: for short-lived, low-consequence interactions, provider-managed state can be the right complete implementation. Reconstructing every turn locally may add cost without useful control. The concession ends when the conversation crosses users, lasts beyond a session, invokes consequential tools, enters background execution, or must satisfy residency and deletion obligations.

The architectural question is not whether a response API is stateful. It is which state it owns, how long that state exists, what evidence the application retains, and what happens when the provider cannot continue the chain. Stateless protocols move responsibility to clients; stateful response APIs move some of it into provider infrastructure. Neither direction removes the responsibility to design continuity explicitly.