Debate and ensembles buy reliability with tokens — price the trade explicitly

Every multi-model reliability pattern — sampling-and-voting, mixture-of-agents, multi-agent debate, heterogeneous model fleets — is the same purchase: redundancy, paid for in tokens. That's a respectable purchase; redundancy is how engineering has bought reliability since N-version programming in 1985. What I keep seeing in practice, though, is teams adopting these patterns for the accuracy points without ever writing down the exchange rate.

The cost structure isn't hidden — the research states it plainly. Mixture-of-agents runs N proposers across L layers, so cost scales as N×L and time-to-first-token stretches with every layer; the MoA papers publish cost–quality Pareto frontiers, and MoA-Lite exists precisely because the full configuration overspends for most workloads. Sampling-and-voting has a scaling law with the shape you'd expect: gains flatten while the tenth sample costs exactly what the first did and buys far less. Debate multiplies all of this by rounds. When someone proposes an ensemble, the first question is not "does accuracy improve?" — it nearly always improves — it's "what does this reliability cost per point, and is there a cheaper seller?"

Usually there is, because what an ensemble actually buys is decorrelated failure — independent-ish replicas voting away each other's mistakes — and decorrelation has cheaper suppliers on the same shelf. A verifier scoring N samples from one model (best-of-N with an outcome reward model) gets much of the benefit at a fraction of the aggregation machinery. Process reward models, which score reasoning step by step, beat outcome-only verification by double digits on hard math — 94% versus 78% on MATH in the canonical comparison — when step-level search is worth its training cost. Cheapest of all is calibrated confidence with escalation: measure when the model knows it doesn't know, route only the uncertain fraction of traffic to the expensive path — ensemble, bigger model, or human — and let the confident majority through single-pass. Conformal methods make that routing threshold a guarantee instead of a vibe. Selective redundancy nearly always dominates blanket redundancy.

Two honest complications keep this from being mechanical. Verifiers can be gamed — reward hacking is inherent to learned reward models, and PRM over-optimization is measurable: beam search under a process reward model degrades on easy problems where plain best-of-N wins. And calibration itself decays — RLHF demonstrably worsens it, so the confidence scores your routing depends on need their own monitoring. The pricing discipline has to include the reliability of the pricing instruments.

The concession: there is a regime where debate and ensembles are legitimately the cheapest reliability available — when single-pass accuracy is low, when verification is as hard as generation so no verifier shortcut exists, and when the cost of a wrong answer dwarfs the token bill. Models are systematically bad at seeing their own errors; a genuinely different model critiquing the answer finds failures self-consistency never will. In those corners — high-stakes review, adversarial settings, frontier-difficulty reasoning — pay for the fleet without guilt.

Everywhere else, the rule is the one that governs all agent economics: nothing is wrong with buying reliability with tokens; what's wrong is buying it unpriced. Define the reliability target per task class, measure single-pass accuracy, then compare the interventions — bigger model, verifier, selective escalation, full ensemble — on reliability per token. It's the cost-is-architecture argument in miniature, applied to the most quietly expensive pattern in the multi-agent playbook.