Streaming isn't UX polish — it's the agent runtime's backpressure design

Streaming is usually sold as a presentation feature: replace the spinner with tokens and the product feels faster. I think that framing is backwards. In an agent system, the visible token stream is only the last consumer in a chain of producers and consumers. The real design question is whether each boundary can slow, pause, cancel, and account for the work upstream. Streaming is the agent runtime's backpressure design; the typing effect is merely its most visible exhaust.

A production agent does not produce one string. It produces a sequence of state changes: text deltas, tool-call starts, partial arguments, tool results, usage updates, completion, interruption, and failure. If those arrive as an undifferentiated byte stream, the application has no reliable place to enforce policy. A typed event protocol turns the stream into a control surface. The runtime can route text to the renderer, hold incomplete tool arguments, charge usage to a budget, mark a tool as in flight, and distinguish a clean finish from a broken connection. Provider-specific events should be normalized at this boundary, because application code should not inherit every SDK rename or wire-level quirk.

Backpressure is what keeps that control surface honest. An async generator naturally pauses at each yield until its consumer asks for the next item. A credit-based protocol makes the same bargain explicit: the consumer grants demand, and the producer must not send beyond it. Across a desktop boundary, the renderer can acknowledge that it processed a chunk before the main process sends another. Without that contract, "streaming" can mean fire-and-forget delivery into an unbounded queue. The model may be generating smoothly while the browser is seconds behind, memory is climbing, and the interface is repainting a past the user can no longer interrupt.

This is also why streaming changes the agent loop rather than merely exposing it. A complete tool call can be extracted and begin executing before the rest of the model response has finished. Independent tools can run concurrently, while their progress events are multiplexed by call ID into one trajectory. Long-running tools can yield partial results, allowing the runtime to truncate output, enforce a budget, or stop work mid-flight instead of discovering the violation after the final return value. Streaming creates scheduling points. Those points are where concurrency, ordering, and resource governance become implementable.

Cancellation is the clearest test. If the Stop button only hides new tokens while the model request and tools continue consuming compute, the product has animation, not streaming architecture. A real cancellation signal propagates through the transport, aborts the in-flight provider request where possible, stops the correct tool streams, preserves already-delivered output as explicitly interrupted, and releases resources in cleanup. The same event lifecycle makes failure legible: sending, streaming, complete, interrupted, and failed are different runtime states, not variations of one boolean.

The stream is an observability boundary too. When events carry identity, timing, usage, and terminal status, the runtime can tell whether a pause came from model generation, a tool, the network, or a slow renderer. It can replay what the user actually saw, in order, rather than reconstructing a final answer that erases partial work and timing. That connects streaming directly to trajectory replay: the execution trace and the client-visible stream should be two views of the same event history.

The concession is precise: not every workload should stream, and not every upstream producer can truly be slowed. Scheduled agents and all-or-nothing jobs are often better served by a job ID plus polling or a webhook. Some model APIs will continue decoding even when a downstream consumer pauses, so backpressure can bound buffers and trigger cancellation without controlling generation token by token. In those cases, say so. Choose asynchronous delivery, bounded queues, or deliberate buffering. But do not confuse replaying a buffered answer with a typewriter effect for flow control.

I now review streaming designs by asking one question: what happens when the consumer stops reading? If the answer is "messages queue up," the runtime has no backpressure design. If the answer names demand, bounds, cancellation, terminal states, and cleanup, the system is not merely pleasant to watch. It is governable while it runs.