The LLM gateway is a control point, not plumbing

Most teams adopt an LLM gateway as plumbing — a proxy that normalizes provider APIs so application code doesn't care whose model answers. That framing undersells the most important property the gateway has: it is the one chokepoint every token in the system crosses. In distributed systems, the chokepoint everyone must traverse is never plumbing. It's where policy executes — and the gateway is the cheapest place in an AI architecture to enforce cost, reliability, and compliance decisions, because it's the only place that sees them all.

Start with routing, which looks like load balancing and is actually policy. A model alias decouples the application from the provider — that much is plumbing. But every strategy behind the alias encodes a judgment: latency-based routing with EWMA tracking says responsiveness outranks cost; cost-based selection says the opposite; learned routers like RouteLLM estimate the probability a strong model beats a weak one per query and route against an explicit cost threshold — a dial that literally prices quality. Tag-based routing pins EU traffic to EU deployments and PII-bearing requests to no-retention providers, which makes the gateway where data-residency law is enforced rather than documented. Whoever writes the routing table is setting product and compliance policy, whatever their job title says.

The reliability machinery says the same thing louder. A production gateway carries typed fallback chains keyed to an error taxonomy — context overflow falls back differently from a rate limit, which fails over immediately rather than waiting out a retry. Cooldowns and circuit breakers quarantine unhealthy deployments. All valuable — and all policy, because a fallback is a decision about what quality degradation you'll accept silently. The silent-fallback quality risk is real: users spend an afternoon on a degraded model and nobody knows, unless routing decisions are logged as first-class events. Provider drift compounds it — model behavior shifts behind unchanged model IDs, and the gateway is the only layer positioned to pin versions and detect the shift.

Then there's the ledger. Per-key and per-team budgets, rate limits, metering, per-tenant guardrail chains and system-prompt injection, semantic caching — each lives at the gateway because each requires seeing every request. This is why cost is an architectural property: the gateway is where the meter runs, and a budget that isn't enforced at the chokepoint isn't a budget, it's a hope. The same logic makes the gateway the natural home of the cache that can lie to you — and the place its thresholds get governed.

The concession: gateways carry a real over-abstraction risk. Normalizing every provider to one API means hiding the capabilities that differentiate them — a new reasoning parameter, a beta header, a provider-specific tool mode — and a gateway without an escape hatch quietly caps your ceiling at the lowest common denominator. It's also a component you must now operate: a shared-state coordination layer that can itself become the bottleneck or the outage. The answer to both is engineering, not abandonment — passthrough options for provider-specific parameters, and the same HA discipline you'd give any control plane.

The practical conclusion is about ownership and change control. If the gateway is plumbing, its config is YAML someone edits when a provider has a bad day. If it's a control point, then routing tables, fallback chains, budgets, and tenant policies are policy code — versioned, reviewed, tested, with routing decisions logged and auditable. Buy one or build one, but govern it like the policy engine it is. The gateway is where your architecture's promises about cost, quality, and compliance either execute or evaporate.