Preference datasets need disagreement topology, not majority labels

I want preference datasets to preserve disagreement topology, not just majority labels. By topology I mean the pattern of judgments across response segments, evaluation criteria, judges, and the evidence available to them. A winning response compresses those relationships into a training target. If the dataset retains only that target, the team loses the ability to explain what the preference meant or rebuild it when the product’s priorities change.

Multimodal alignment makes this loss particularly visible. The multimodal preference alignment hub describes language priors overriding visual evidence and a tension between helpfulness and visual faithfulness. Imagine a detailed answer that invents an object beside a cautious answer that describes only what is visible. Reviewers could disagree because they value completeness differently, because they missed the invented object, or because the image is ambiguous. The same vote split can represent different problems. I would not route them through the same correction policy.

RLHF-V gives me a concrete reason to preserve the unit of judgment. Its segment-level corrections and masked preference loss localize the contrast between a hallucinated passage and its correction. A whole-response ranking cannot express that distinction by itself. I would retain the original response, the corrected span, and their relationship rather than treating the corrected answer as an unrelated winner. This preserves which claim needed repair without implying that every sentence in the rejected response was wrong.

LLaVA-RLHF exposes another dimension — the judge’s evidence. Its factual augmentation supplies reward-model context rather than assuming a preference judgment already contains sufficient grounding. My architectural inference is that annotation context belongs in the dataset lineage. I want to distinguish judgments made from an image alone from judgments supported by reference descriptions. Otherwise, a later disagreement analysis can mistake unequal access to evidence for a difference in values or reviewer competence.

Silkie’s multidimensional feedback and model-pool construction point toward a richer record than a preferred-answer column. I would store criterion-level judgments, candidate-model provenance, judge identity or version, and the annotation recipe as related records. Human and model judgments should remain distinguishable. Agreement with a human sample does not make a model judge interchangeable with every future reviewer. The policy that combines those judgments should have its own version, separate from the observations it consumes.

POVID adds a provenance distinction that I would make mandatory — a rejected response created through hallucination injection or visual deprivation is a constructed negative, not evidence of human dissent. Those methods provide targeted contrasts for training. Their lineage tells us what failure mechanism the contrast was designed to expose. Mixing constructed negatives with independently judged comparisons without preserving that distinction would make the apparent preference distribution difficult to interpret. Dataset scale would conceal a change in what the rows actually represent.

I would implement disagreement topology as queryable relationships, not insist on a graph database. The practical questions are ordinary data-platform questions: which criteria reverse a ranking, which spans attract corrections, and which judge versions disagree on the same evidence? Training pairs would be derived views over those records. Evaluation slices would follow the same dimensions. That makes aggregation a reproducible policy decision and gives the next annotation batch a reason to target unresolved cases rather than collect more easy agreement.

I concede a narrow case: majority labels can be sufficient for a bounded comparison task when reviewers share the same evidence, apply a stable rubric, and audits show that residual disagreement is incidental error rather than a recurring criterion-level split. In that setting, a simple training export is reasonable. I would still preserve the underlying judgments long enough to test whether that boundary continues to hold as candidates and reviewers change.

My claim is a data-design recommendation drawn from these alignment mechanisms, not a claim that they establish disagreement topology as a benchmarked method. I want the durable asset to be the evidence behind preference — with the winning label treated as a replaceable projection. When a model learns the wrong trade-off, the platform should let us inspect and revise that projection. A majority-only archive leaves us with a decision and no adequate account of how it was made.