Cost is an architectural property

Your LLM bill is not set by your provider's price sheet. It is set by decisions your architects made months before the invoice arrived. Treating cost as a procurement problem — negotiate the rate, cap the budget, escalate when finance complains — is how teams end up bolting on controls after the spending pattern is already baked into the system's structure.

The evidence is in the gap between quoted and effective price. The cost-engineering literature I've built my knowledge base around documents a roughly 9x gap between token price and effective cost per useful result — the difference is architectural waste: retries that re-send full context, ReAct loops that re-inflate the entire history on every iteration, RAG pipelines that stuff ten chunks where two would do. This is the prototype illusion: the demo costs pennies because it runs one clean pass, and nothing in the per-token price warns you that your loop structure multiplies it. The number that matters is cost per successful task completion, and that number is a function of design, not of rate cards.

Four decisions that set the bill

First, caching strategy is prompt structure. Anthropic's prompt caching bills cache reads at 10% of input price with break-even after a single reuse — but only if your prompt keeps static content (system prompt, tool definitions, documents) first and dynamic content last, because the cache key is a byte-exact prefix. Inject a timestamp into the system prompt or reorder tools, and every request is a cache miss. A single stray whitespace change breaks the match. You cannot fix this in procurement; it's the shape of your prompt assembly code. Semantic caching adds a second layer — GPTCache-style meaning-keyed lookup returns near-zero-marginal-cost answers for paraphrased queries — but its threshold calibration is a design decision with a failure mode (false positives serving factually wrong cached answers), not a toggle.

Second, context discipline. Tool schemas, XML tags, and formatting overhead alone consume 20–40% of the tokens in a typical agent turn. Compaction policy, truncation strategy, and what survives into each turn are architecture — and they interact with caching, because a compaction that rewrites the stable prefix busts the cache in the same stroke that shrinks the context.

Third, model routing. The cost–quality Pareto frontier is real: fine-tuned small models handle well-scoped tasks at a fraction of frontier-API cost, and per-key model aliases in a gateway let you tier traffic without touching application code. Routing everything to the flagship model is a default, not a decision — and defaults are the most expensive architecture of all.

Fourth, gateway budgets as load-bearing infrastructure. A LiteLLM-style virtual-key hierarchy — provider key, master key, virtual key, with budgets cascading global to team to key, per-session spend caps, and iteration limits on agent runs — is the difference between "we noticed the overrun in the monthly invoice" and "the run was stopped at its budget." Even here architecture bites: asynchronous spend increments open an overspend window where enforcement lags actual spend. If budgets are enforced only in a dashboard, they are decoration.

The honest counter-case: for a low-volume internal tool, none of this earns its complexity. If the workload is a few thousand calls a month, the entire annual bill may be smaller than the engineering time a caching layer costs, and "just pay the rate card" is the correct architecture. Cost design is proportionate to volume — the failure I'm naming is not skipping it at small scale but discovering at large scale that it was never designed at all, which is one face of the demo-to-production gap.

The discipline is simple to state: model your unit economics — cost per successful task, not cost per token — at design time, the same week you choose your framework. Every architecture has a cost curve. The only question is whether you drew it on purpose.