Alternative sequence models should be chosen by state-update economics, not context length

I would choose an alternative sequence model by the economics of updating its state, not by the largest context length on its card. Context length describes an admissible input; it does not describe what every new token forces the machine to retain, read, rewrite, or discard. For production architecture, the useful unit is the state transition—its bytes moved, operations performed, memory tier touched, and information lost. This is one more case where cost is an architectural property, not a bill discovered after model selection.

Self-attention makes history explicit through a growing key-value cache. That preserves direct access to prior tokens, while the serving system pays for state that expands with the sequence. State-space and recurrent designs make a different bargain: they compress prior inputs into a fixed-size inference state and update it token by token. The headline changes from “how much context fits?” to “what does one update cost, and what evidence survives it?” That is a sharper comparison because it joins latency, memory capacity, and recall in the same question.

The state update must be priced against the memory hierarchy. A mathematically linear recurrence is not automatically cheap when its implementation repeatedly crosses the SRAM/HBM boundary, exposes little parallel work, or moves more state than the arithmetic justifies. Mamba’s selective scan, parallel-scan techniques, and hardware-aware algorithm design matter because the operator and the machine are being designed together. The relevant benchmark is therefore not asymptotic notation alone. It is the realised update path on the target hardware—especially during autoregressive decoding, where state movement can dominate useful calculation.

Training economics belong in the same decision. Recurrence–convolution duality allows related state-space computations to present one form for parallel training and another for recurrent inference. RWKV similarly makes dual training and inference modes part of the architecture, while Mamba-2 uses structured state-space duality and chunked computation to reorganise the work. These are not implementation footnotes. They determine whether the model can use parallel hardware efficiently while learning and still carry compact state while serving. A model that wins only on one side may simply move cost across the lifecycle.

State quality is the second half of state price. Selective SSMs make updates depend on content; boundary resetting gives the model a way to stop irrelevant history leaking across segments; Mamba’s delta parameter changes how inputs influence state evolution. RWKV’s time decay and receptance make related choices about retention and access. Each mechanism spends computation to decide what should persist. That makes prompt sensitivity and selective-copying or induction-style evaluations operational evidence: they test whether cheap state updates preserve the information the workload actually needs.

This reframes architecture selection as a workload table. For each candidate, I want the inference-state size, bytes moved per generated token, update latency at realistic batch sizes, training form, chunking strategy, and recall behaviour across relevant boundaries. I also want the placement target, because on-device models redraw the privacy and latency boundary and change which memory hierarchy matters. The method resembles choosing OLAP engines by query shape and operating model, not benchmark rank: fit is produced by the workload and the operator, not a universal leaderboard.

One precise concession: when a workload requires frequent, exact retrieval of arbitrary earlier tokens and has enough memory bandwidth to sustain the growing cache, attention’s explicit history can be economically superior to compressing that history into recurrent state. That boundary is specific and testable. It is the efficiency–recall trade-off expressed as an operating condition, not a reason to treat every long-context workload as identical.

The practical choice is rarely “Transformer or Mamba?” in the abstract. Hybrids such as Jamba, Zamba, Hymba, Samba, and Griffin exist because different layers can buy different kinds of access, retention, and throughput. I would benchmark whole state-update regimes—including hybrid ones—at the sequence lengths, batch shapes, and hardware tiers the system will actually use. Then I would gate regressions in that unit, just as cost regression gates belong beside eval gates. Context length is a capacity claim. State-update economics tells me whether the architecture can afford to use that capacity repeatedly.