LLM gateway as the control plane for agent systems

The gateway is not a provider proxy. It is the runtime chokepoint where cost, quality, privacy, residency, and reliability policy become one executable routing decision.

Problem

Most teams introduce an LLM gateway to normalize provider APIs. That removes SDK duplication but leaves the important decisions scattered: one service chooses a model, another tracks spend, an eval dashboard says whether the model is acceptable, a security filter redacts prompts, and retry middleware silently changes providers during an incident. Each control may work locally while the request as a whole becomes impossible to explain.

An agent request needs a stronger invariant. Before it reaches any provider, the system should be able to answer: which policy version governed it; which models were eligible; which candidates were rejected for capability, residency, health, or quality; whether the remaining budget admitted it; what data was transformed; whether fallback changed the route; and what the successful call actually cost. If those are separate middleware facts assembled after the event, the gateway is still plumbing.

Constraints

The reference build had to make policy composition visible without hiding behind a real proxy or provider SDK. Provider behavior therefore had to be deterministic and offline. A cheaper model could not bypass an evaluation floor. A fallback could occur only for a declared retryable failure before output began — never for policy denial, semantic failure, or a response already in flight. Spend had to be checked before dispatch and debited from actual usage after success. Sensitive values could not reach provider adapters or the evidence ledger. Finally, one request had to remain bound to one immutable policy version even if a newer snapshot became active during execution.

Decisions

1. One snapshot governs the complete request. The gateway binds an immutable, versioned policy object at ingress and passes the same value through authorization, redaction, routing, budget, fallback, and evidence. A concurrent activation changes future requests, not a request already in flight. This turns gateway configuration into the versioned policy code argued for in the LLM gateway is a control point, not plumbing.
2. Eligibility precedes optimization. Candidate models must first satisfy required capabilities, requested residency, provider-health state, and an approved per-model evaluation floor. Only eligible targets are ranked. Cost and latency optimize inside the approved set; they cannot convert an unqualified model into an acceptable one.
3. Budget is admission control. Before dispatch, the gateway estimates the maximum cost across the eligible fallback chain and denies the call if that reservation exceeds remaining daily spend. After success it debits actual adapter-reported token usage. This makes the runtime budget the last line of defence described in cost regression gates belong in CI next to eval gates.
4. Fallback is a typed policy decision. Every provider failure carries an error class and an output_started marker. The gateway advances only when the policy declares that class retryable and no output has begun. The resulting degradation is explicit in the outcome rather than silently disappearing behind a stable alias.
5. Evidence records the route, not the prompt. The SQLite decision ledger stores the policy version, redaction action names, prompt digest, target exclusions, route attempts, resolved model, simulated latency, usage, cost, outcome, and degradation marker. Raw prompts and matched sensitive values are excluded. A SHA-256 hash chain makes mutation detectable.

Outcome

The deterministic demo sends an EU agent request for the logical alias agent-balanced with tool-use capability. The gateway redacts an email and API token before dispatch. It rejects a cheap model below the evaluation floor, a US target outside residency, and a text-only target without tools. Budget admission reserves the maximum eligible request cost. The preferred EU provider then returns a typed retryable pre-output failure, so the explicit fallback succeeds and the result is marked degraded.

The successful fixture reports 18 input tokens, 6 output tokens, 45 ms simulated latency, and an actual calculated cost of 0.00006. Those are deterministic test fixtures, not a benchmark or savings claim. The ledger replays the complete decision and verifies its hash chain.

The artifact includes ten behavioral tests, strict lint and type checks over thirteen source files, a verbose offline demo, and seven self-checking scenarios covering budget denial, quality-over-cost, residency and capability gates, provider failover, privacy redaction, immutable policy binding, and ledger tamper detection. The full source is public at github.com/Dhristhi/llm-gateway-control-plane.

What you can run

git clone https://github.com/Dhristhi/llm-gateway-control-plane
cd llm-gateway-control-plane
uv sync --extra dev
uv run pytest -q
uv run python -m gateway_demo.demo
for s in scripts/scenario_*.py; do uv run python "$s"; done

The honest caveat: this validates policy composition and decision evidence, not proxy throughput, real-provider compatibility, streaming transport, distributed quota consistency, secure secret discovery, production latency, or availability SLOs. The budget tracker is in memory, token estimation is deliberately simple, SQLite is a validation surface, and deterministic adapters stand in for provider clients. A production gateway would need atomic distributed reservations, streaming-aware failure boundaries, external policy authorization, real tokenization, signed ledger checkpoints, and measured provider behavior. The narrower claim holds: the gateway can make an agent request's cost, quality, privacy, and reliability policy execute as one explainable decision.