Human feedback is a control loop — unmeasured disagreement becomes hidden bias

Human feedback is not a neutral stream of labels — it is a control loop that determines which behaviours an agent learns to repeat. I distrust any feedback programme that reports only an average preference score or a single “human-approved” label. When annotators disagree, that disagreement is information about ambiguity, competing values, uneven expertise, or different exposure to harm. Collapsing it into one target does not remove the conflict. It hides the conflict inside the reward model, where a majority preference can later look like objective system behaviour.

The control-loop framing changes what I measure. An agent produces a trajectory; people inspect the result, the intermediate choices, or both; their rankings, edits, scalar scores, and natural-language critiques become a training signal; the updated agent then changes the trajectories people see next. Each pass alters the distribution of future evidence. Online and streaming feedback make the loop faster, but speed also amplifies whatever the measurement design ignores. A team that records only the winning answer cannot tell whether agreement was strong, whether reviewers split into stable camps, or whether one group’s objections disappeared through aggregation.

This matters more for agents than for single-turn assistants because preference can attach to several levels at once. Two reviewers may accept the same final answer while disagreeing about a risky tool call in the middle. They may prefer different complete trajectories even though both appear successful. Step-level, segment-level, and trajectory-level annotation therefore answer different questions. I want the feedback record to preserve who evaluated what unit, which criterion they applied, their confidence, and the distribution of judgments — not merely the label that survived adjudication.

Fairness work gives the warning in sharper terms. Aggregation bias appears when groups with different distributions are averaged into a common rule; evaluation bias appears when the people or benchmarks doing the judging do not resemble the population that will experience the system. Even the formal definitions of fairness conflict when base rates differ, so “align with human preferences” cannot be a complete requirement. Whose preferences, over which harms, under which operating conditions, and with what trade-off? A reward function that omits those questions does not become impartial. It turns unstated governance choices into optimisation targets.

I would retain disagreement as first-class telemetry. For every preference query, I would store the full vote distribution, annotator cohort and relevant expertise, criterion-level scores, confidence, and adjudication path. I would monitor disagreement rates by task type and affected group, then use uncertainty and disagreement to select the next cases for review. That turns active preference learning into more than sample efficiency — it becomes a way to seek the boundary where the current policy is least socially or operationally settled. I would also compare slices rather than trust an aggregate win rate, because stable overall performance can coexist with a systematic loss for a small cohort.

Human-agent collaboration adds a practical consequence: feedback quality depends on whether people remain capable of giving it. Long autonomous runs create an out-of-the-loop problem; repetitive approvals create rubber-stamping; opaque trajectories leave reviewers judging outcomes without understanding how they were produced. Structured briefings, trajectory replay, batched updates, and reversible checkpoints are therefore parts of the learning system, not interface polish. They preserve the reviewer’s situational awareness and make correction specific enough to repair the agent’s world model rather than merely veto its latest action.

There is one precise concession — if the task is deterministic, low-stakes, fully specified, and scored by an execution oracle, annotator disagreement may add little because correctness already supplies the control signal. Most production agent work does not meet that boundary: goals are incomplete, actions have side effects, and reasonable people weight risks differently. There, disagreement is not annotation noise to smooth away.

I would treat alignment dashboards the way I treat production observability: averages are entry points, not explanations. Show me consensus strength, dissent clusters, cohort-specific error rates, changes after each policy update, and whether the next feedback batch covers the regions where reviewers diverged. Then the loop can expose its own blind spots. Without those measurements, “human feedback” becomes a reassuring name for a hidden policy — and the bias is hardest to challenge precisely because the system has learned to call it consensus.