Privacy budgets are runtime resources, not compliance numbers

A privacy budget is a runtime resource, not a compliance number written beside a model. In a long-running agent, every private retrieval, aggregate, or generated response can spend part of the protection promised by differential privacy. If the runtime cannot allocate that spend, compose it across calls, refuse work when it is exhausted, and explain who consumed it, then the stated epsilon is not a control. It is an annotation on an unbounded process.

The online composition problem makes this unavoidable. A one-shot analysis can choose a mechanism and account for its privacy loss once. An agent can query memory repeatedly, reformulate after a weak result, delegate to another agent, and revisit the same population next week. Each individually acceptable query changes the remaining envelope. Recurrence is therefore not free evidence; it is cumulative disclosure risk. The runtime needs a ledger whose unit is the protected subject or dataset, not merely the API key that submitted the request.

I would model that ledger like any scarce resource. Admission reserves an estimated privacy cost before execution. Completion reconciles the reservation against the mechanism actually used. The record carries purpose, sensitivity class, dataset version, mechanism, accounting method, requesting principal, and the parent run that authorized it. A child agent inherits a bounded allocation rather than ambient access to the parent’s whole budget. Parallel branches reserve atomically, because checking the balance and spending later creates the same oversubscription window that makes a supposedly hard financial limit soft.

Sliding-window accounting adds lifecycle semantics. A rolling window can make older contributions expire, but expiry is meaningful only if the underlying data and derived artifacts follow the same retention promise. Removing yesterday’s epsilon from a counter while keeping the synthetic corpus, cached answer, or memory summary it purchased is accounting theatre. This is where privacy becomes a data-flow problem: budget renewal depends on deletion, provenance, and derived-data lineage, not on resetting a dashboard at midnight.

Allocation should also be personal and purpose-aware. Sensitive interactions may receive tighter limits; low-risk aggregates may use a different envelope; one user should not silently spend another user’s protection through a shared tenant pool. The scheduler then has real choices when capacity runs low: degrade to a non-private-data path, answer from an already approved aggregate, ask for narrower scope, defer the task, or stop. That is the privacy equivalent of making budget exhaustion a first-class terminal state rather than letting the model improvise around a denied call.

Cryptographic privacy mechanisms widen the toolbox without removing the accounting problem. Trusted execution environments can protect data during processing, secure multi-party computation can divide trust, and zero-knowledge proofs can verify selected claims. Those controls answer who can see or verify a computation. Differential privacy answers what repeated outputs reveal about contributors. A production design may combine them, but it should not confuse attested execution with bounded disclosure.

There is one precise concession: a fixed, one-off release over a frozen dataset can reasonably calculate its privacy loss before execution and treat the budget as deployment configuration. The claim becomes a runtime claim when queries are interactive, recurring, delegated, personalized, or composed across services. That is exactly the operating shape of an agent.

I want privacy budgets exposed to schedulers, policy engines, traces, and incident review with the same seriousness as money, tokens, or compute. Placement constrains where data may exist; memory governance constrains what may persist; privacy accounting constrains how much evidence the system may release over time. The number matters. The machinery that spends and stops it is what makes the number true.