Autoscale agent fleets on queue age and checkpoint cost, not CPU

Autoscaling an agent fleet on CPU is measuring the worker while ignoring the work. Most agent runtimes spend long intervals waiting on model APIs, tools, approvals, and remote systems. A pod can be nearly idle while its queue grows old enough to violate a customer promise. It can also be busy checkpointing a valuable long-running task at exactly the moment a conventional autoscaler decides to remove it. I would scale agent capacity from work debt — queue age, service deadlines, dependency headroom, and interruption cost — not from container utilization alone.

Queue depth is a useful start, and KEDA makes the mapping explicit: choose how many messages one replica should own, divide queued work by that target, and derive the desired replica count. But raw depth assumes all messages are interchangeable. Ten quick classifications and ten research runs with external tools are not the same backlog. The signal I care about is the age distribution of eligible work, weighted by its service objective and estimated remaining cost. An old high-priority case is debt even when the queue is short; a large batch with no deadline may not justify an emergency scale-out.

Dependencies put a hard ceiling on how quickly that debt can be repaid. If the model provider is saturated or a downstream API is rate-limiting, adding workers converts queue pressure into retry pressure. The autoscaling research’s composite-trigger pattern is the right instinct: combine demand with an LLM latency or quota signal so scale-out occurs only while the constrained dependency has headroom. Otherwise the fleet grows exponentially around a bottleneck it cannot move, raising cost and amplifying failure without improving completion time.

Scale-in is the more neglected half. Agent work is often stateful across minutes or hours. Kubernetes preemption knows pod priority but not the value of the reasoning, tool results, and approvals held inside a run. Graceful termination helps only if the runtime can turn SIGTERM into a durable checkpoint before the grace period expires. I therefore treat checkpoint cost as a scheduling input: prefer removing idle workers, then workers between steps, and only then active workers whose state is cheap to persist and resume. Killing an expensive run to save a small amount of idle capacity is negative optimization.

This also changes the choice between long-lived workers and per-message jobs. A ScaledObject suits a queue of many short tasks because warm workers amortize startup and process repeatedly. A ScaledJob buys process isolation for each message, but pod startup and image pulls become part of every task’s latency. Long-running workflows belong in a durable engine or a worker that externalizes state, not in an ephemeral job whose completion semantics hide hours of recoverable progress.

There is a boundary where CPU remains the right signal. For locally hosted inference or compute-heavy tools with predictable request cost, utilization directly represents the scarce resource, and queue age can complement rather than replace it. My claim is about API-bound, tool-using agent fleets, where CPU is usually a weak proxy for demand and an even weaker proxy for customer pain.

The operating dashboard I want therefore shows oldest queue age, deadline risk, dependency saturation, checkpointability, and cost per completed task. Those measures connect capacity decisions to outcomes. CPU still belongs on the dashboard as a substrate health metric. It just should not be allowed to decide how much agent work the platform can responsibly accept, interrupt, or promise to finish.