Cost regression gates belong in CI next to eval gates

Cost regression gates belong in CI next to eval gates. If a pull request can be rejected because it makes an agent less accurate, it should also be rejectable because it makes the same successful task materially more expensive. Quality and cost are not separate review streams. They are two dimensions of the same production contract.

The placement matters. A monthly invoice can tell you that spend rose; it cannot tell you which prompt edit, tool description, routing change, or retry policy caused it. By then several changes may have accumulated and the cheapest moment to question them has passed. CI has the causal context the invoice lacks: this change, against this versioned eval set, produced this quality delta and this usage delta. Put both in the pull request and the trade becomes reviewable before it becomes architecture.

The implementation is less exotic than the governance language makes it sound. Run the candidate and the approved baseline over the same eval cases. Capture input, output, and cached-token usage from every model call, plus the model and prompt version. Compare cost per successful task, not merely total tokens: a cheaper run that fails more often and retries in production is not cheaper. Report the suite-level delta, then break it down by case and by component — system prompt, retrieved context, tool definitions, model route, retries — so the gate explains the regression instead of merely announcing it. A common first finding is that tool descriptions, not the core prompt, are silently driving cost; that is a schema detail no one was asked to review. The same artifact that feeds the quality gate can feed the cost gate, which is why the two sit side by side.

I would use two policies together. An absolute budget says an operation must remain economically viable; a relative gate says this change must not drift beyond its approved baseline. The baseline should be the last deliberately approved state, stored with the eval dataset and model configuration, not a number silently refreshed after every run. When the gate fails, the pull request should show the quality gain beside the cost increase and require an explicit decision: optimize, reject, or approve the new baseline with a reason. That is not bureaucracy. It is the cost equivalent of accepting a larger frontend bundle because the feature is worth the load-time penalty.

The important concession is that cost gates are noisier than deterministic tests. Output length varies, provider pricing changes, cache behaviour differs between CI and production, and model routing can make identical inputs take different paths. A naive hard threshold on one run will create false alarms; a tiny team with low-volume internal usage may spend more engineering time maintaining the gate than the gate can save. In those cases I would start with an informational check, use repeated runs or confidence bands where variance warrants them, and promote it to a blocking gate only when the workload and measurement are stable. But measurement noise argues for calibrated gates, not for discovering regressions in finance reports.

Nor should CI pretend to be the whole control system. Per-PR gates catch step changes; immutable trend history catches the boiling frog of many individually approved increases; production metering catches traffic mix, retries, and cache-hit rates the test suite cannot reproduce. Runtime budgets at the LLM gateway remain the last line of defence. These layers answer different questions: did this change raise unit cost, has unit cost drifted over time, and is live spend still inside policy?

This is why I think cost gates belong beside eval gates rather than in a FinOps annex. Evals define what the system must accomplish. Cost regression gates define the resource envelope within which that accomplishment remains a product. Once prompts are source code and cost is an architectural property, an ungated cost increase is simply an unreviewed production regression.