Voice agents are latency budgets with a language model inside

A voice agent is not a chatbot with a microphone. It is a hard real-time system with an end-to-end latency budget of roughly 245–565 milliseconds — target under 400 — and a language model sitting in the middle of it, spending most of that budget. Once you see it this way, every architecture decision in a voice system becomes legible as a spend against the budget, and most production failures become legible as overspends.

Start with the pipeline shape. The dominant pattern is still the STT–LLM–TTS cascade: transcribe, think, speak, each stage streaming into the next. Native speech-to-speech models collapse the stages; half-cascade hybrids take a realtime model's audio input but keep a separate TTS stage precisely because it gives back control over the most audible part of the budget. The choice between these isn't a quality argument first — it's a question of where you can pipeline. Overlapped and speculative execution, where the LLM starts generating before end-of-turn is fully confirmed and TTS starts speaking before the LLM finishes, is the voice equivalent of instruction pipelining, and prompt caching alone can cut LLM time-to-first-token by 70–90%. None of this improves the answer. All of it buys back milliseconds.

Turn-taking is a policy, priced in milliseconds

The budget thinking gets sharpest at turn detection. Deciding that a person has finished speaking is a layered inference — voice-activity detection, ASR end-of-utterance signals, semantic "is this thought complete" classification — and every layer trades accuracy against waiting time. Production systems now treat the trade as an explicit policy: a strict end-of-turn policy waits 800–1200ms of silence plus semantic completeness, which is right for clinical or financial conversations where cutting someone off is a real cost; a progressive policy responds on partial transcripts and commits after a 300–500ms pause, which is right for support and IVR traffic. The same is true of interruption. Barge-in is not a toggle; distinguishing an "uh-huh" backchannel from a genuine interruption is a hard classification problem, the recovery window for false positives is itself a tuned parameter, and analysis across millions of production calls keeps finding the same named failure modes — the whisper, the TV in the background, the cancel-restart loop. These are engineering artifacts with telemetry schemas, not prompt-tuning problems.

Design the wait, don't just shrink it

The budget also has a UX escape valve: when a tool call genuinely takes three seconds, the answer isn't a faster model, it's a filler utterance — "let me check that" — that spends conversational naturalness instead of compute. The 2026 production default is hybrid for the same reason: on-device reflexes handling acknowledgment and interruption in under 300ms while cloud reasoning handles the thinking. And evaluation follows the frame — time-to-first-audio, barge-in fidelity, and word error rate are budget metrics, measured per turn, next to the coherence scores that judge the language model itself.

The honest concession: native speech-to-speech realtime models are getting good fast, and they collapse much of the budget arithmetic I've just described — when one model hears and speaks, the cascade's stage-by-stage accounting partly dissolves. But the policy layer doesn't. Turn-taking strategy, barge-in rules that vary by what the agent is saying, silence handling, the decision of what runs at the edge — those survive every pipeline shape, because they're properties of conversation, not of the stack. Voice is where agent engineering meets hard real-time discipline, and the teams that ship durable voice agents are the ones that write the budget down first and make the model live inside it.