RL-trained harnesses need policy-version boundaries around tool use

RL-trained harnesses need explicit policy-version boundaries around tool use. Once a model has learned when to call a tool, which argument shape to choose, or when to consult external state, the harness is no longer passive infrastructure. Its conventions are part of the learned policy. If those conventions change during an episode or disappear from the training record, I can no longer say which policy produced the action—or whether the reward belongs to the policy I intend to update.

The boundary starts with a simple observation: tool use defines an action space. Names, descriptions, schemas, permissions, response formats, and error semantics all shape what the agent can do and how it interprets the result. Adding a tool is not merely an integration change; renaming an argument is not merely API maintenance. Either can change the decisions available to the agent. An RL pipeline that versions model weights but treats the tool surface as live configuration preserves only part of its behavior policy.

I would therefore assign every rollout a policy identity that binds the model checkpoint to the harness policy, prompt and context rules, tool-manifest digest, permission policy, and reward-verifier version. That identity should be fixed when the episode begins and carried on every model request, tool call, observation, and reward event. A later operator should be able to read one trajectory and recover the exact decision surface the agent saw—not infer it from whatever the tool server advertises today.

Episode boundaries are the cleanest promotion boundary. A new harness policy can receive new episodes while in-flight episodes finish under the previous version. If an urgent revocation makes that unsafe, the runtime should terminate or pause affected episodes explicitly rather than silently changing their tools midway. A drain requires explicit lifecycle handling; it preserves a coherent relation between observations, actions, and rewards. Quiet mutation creates a trace that looks valid while describing no policy that ever existed as a stable unit.

The same discipline belongs in trajectory reconstruction. Rich traces include tool definitions, calls, responses, intermediate context, and branch structure; trainers consume a narrower sequence of observations, actions, and rewards. Conversion between them must preserve not only the sampled model tokens but also the policy version governing each tool decision. Harness-inserted material should remain distinguishable from sampled behavior, and a schema introduced after rollout should never be substituted during replay. Fidelity is about the whole decision boundary, not text alone.

Version boundaries also make mixed data legible. A training batch may contain episodes from several released policies, but each episode should remain internally coherent and queryable by policy identity. The learner can then filter incompatible traces, compare policy generations, or apply an explicitly chosen off-policy method. Without that separation, an apparent improvement may come from a wider tool set, a different verifier, or changed error handling rather than from the learned update. The provenance problem becomes an evaluation problem before it becomes a governance problem.

A ledger-backed environment makes this implementable. Opening a bounded ledger slice can establish the policy identity at reset; advancing through entries can reproduce recorded actions and observations; the recorded outcome can supply the retrospective reward. Branches created by subagents or parallel calls do not need to be flattened into one fiction, provided every event carries causation, episode identity, and policy identity. The ledger then supports replay and rollback because it records which contract governed the action, not merely that a tool call occurred.

There is one precise concession: a read-only exploratory episode in a disposable sandbox can tolerate an unpinned tool surface when it cannot reach credentials, durable state, private data, or external side effects, and when its traces are excluded from policy updates. That exception is useful for discovery. It should not leak into training or production, where live schema drift would change the action space without changing the recorded policy identity.

I treat policy promotion, tool promotion, and verifier promotion as coordinated release operations. Candidate versions should run in shadow or isolated evaluation, produce separately tagged trajectories, and cross an explicit promotion gate before receiving new episodes. Rollback should restore the whole bound set, not only the model checkpoint. RL makes harness behavior adaptive; version boundaries keep that adaptation attributable. Without them, the system can still learn—but I cannot reliably tell what it learned from, what it is now executing, or what I need to reverse.