On-device models redraw the agent privacy and latency boundary

On-device inference has quietly crossed from demo to usable, and that changes the shape of agent architecture. The interesting question is no longer "can a model run on the phone?" — it's where the boundary between device and cloud should sit for each function of the agent. That boundary is an architecture decision, and the two forces that move it are latency and data sensitivity, not model benchmarks.

What changed is the whole stack, not one model. The 1–7B class got legitimately good — Llama 3.2's edge variants, Phi-4 Mini, SmolLM2, Qwen's edge series — and quantization matured to the point where rotation-based 4-bit schemes like SpinQuant hold accuracy loss under 1%. Speculative decoding buys 2–3.6× throughput on edge hardware. Runtimes caught up too: MLC-LLM compiles models to a sub-5MB binary and pushes 20+ tokens/second on a fanless laptop, and Meta's ExecuTorch went GA for iOS and Android. The classic four motivations — latency, privacy, cost, offline — stopped being aspirations and became purchasable properties.

Here's the part most hardware marketing obscures: TOPS is not the constraint. LLM decode is memory-bound, and mobile silicon moves 50–200 GB/s of LPDDR bandwidth against a server GPU's 2–3 TB/s. A 7B model at INT4 is roughly 3.5GB of weights that must stream through for every token — at 100 GB/s that is a hard ceiling no NPU rating overcomes. This one number explains the whole edge design space: why 1–7B INT4 is the honest sizing, why small models go deep-and-thin, why speculative decoding pays so well. When you evaluate an on-device plan, do the bandwidth arithmetic first; the TOPS comparison is a distraction.

Now the boundary itself. The latency boundary is the easier one to see: reflexes belong local. The pattern proving out in voice systems — sub-300ms acknowledgment and interruption handling on-device while cloud models do the thinking — generalizes to any agent with a real-time surface. The privacy boundary is the deeper redraw. Personal context that never leaves the device is a categorically different governance object from the same data in a cloud memory store: nothing to breach centrally, less to disclose, a far shorter compliance story under data-protection regimes. For an agent that watches your screen, reads your messages, or accumulates behavioral memory, "the model comes to the data" isn't a performance optimization — it's the difference between an architecture you can defend to a regulator and one you can't. Which means agent functions should be partitioned by latency class and data sensitivity, the way a schema partitions by access pattern — not by whichever model happens to be strongest this quarter.

The concession: frontier reasoning stays in the cloud for the foreseeable future. A 3B model will not plan your data platform migration, and pretending otherwise produces on-device demos that fail exactly like cloud demos do — at the hard parts. The boundary moves; it doesn't vanish. Hybrid is the default architecture, all-local is a niche, and the design skill is knowing which functions genuinely earn local placement.

The teams that get this right will treat placement as a first-class design table: each agent capability annotated with its latency budget, its data classification, and its minimum viable model size — then let the bandwidth arithmetic and the governance requirements, not the marketing, decide what runs where. The boundary is a design artifact now; draw it deliberately.