Annotator Disagreement as a Signal in RLHF Pipelines

RLHF pipelines run on a quiet assumption: when two annotators disagree about which response is better, one of them is wrong, and averaging or voting your way to a single label recovers the truth. That assumption is false often enough to matter. Annotator disagreement, the empirical record now shows, is frequently not noise around a hidden consensus but a record of genuine differences in what people value, how they read an ambiguous prompt, or which of several defensible answers to a subjective question they happen to prefer. A pipeline that treats disagreement as something to clean up before training a reward model is not correcting for error. It is deciding, quietly and by default, which human preferences get to survive into the optimization target.

The standard RLHF setup has three stages: annotators compare pairs of model responses and say which one is better, a reward model learns to predict those preferences as a single scalar score, and a policy gets optimized against that score. Each stage compresses. The scalar reward model, in particular, is built to output one number per response, which is a structural mismatch for something as stochastic and heterogeneous as human judgment. Majority voting, averaging across annotators, and filtering out high-disagreement examples all get treated as quality control. They are better described as information destruction, and the distinction matters for anyone trying to understand why a policy trained this way sometimes behaves in ways no annotator actually asked for.

What the empirical record says about where disagreement comes from

Zhang, Wang, Hwang and colleagues, in "Diverging Preferences: When Do Annotators Disagree and Do Models Know?"Zhang, Wang, Hwang and colleagues, in "Diverging Preferences: When Do Annotators Disagree and Do Models Know?", found that more than 30% of examples across both datasets they studied show diverging preferences across annotators, close to a third of the data a reward model is trained on. That is close to a third of the data a reward model is trained on. It is close to a third of the data a reward model is trained on.

More telling is what drives that disagreement. Over 75% of it traces to identifiable factors: response complexity, verbosity, or prompts that leave real room for multiple valid readings. That is a structured pattern, not random error scattered across the dataset. One of the datasets involved, MultiPref, collected 10,000 preference pairs, each with four separate annotator judgments, which is exactly the kind of design that lets researchers see disagreement clearly instead of averaging it away before anyone gets to look at it.

A construct-validity problem that annotation instructions rarely solve produces a lot of this. Ask four people whether a response was "helpful," and one is scoring for information density, another for emotional tone, a third for whether the answer gave them something they could act on immediately. They are answering different questions that happen to share a label. They are answering different questions that happen to share a label. Call this measurement non-invariance: the word "helpful" does not point to one construct, and disagreement here has nothing to do with preference pluralism. It is a labeling problem dressed up as a preference problem.

A working taxonomy of disagreement sources

Zhang et al. sort disagreement into ten categories, organized under four higher-level classes, and the grouping is useful because each class calls for a different fix rather than one blanket aggregation rule.

Task underspecification sits at one end. The prompt genuinely supports more than one valid interpretation, so annotators are not making an error when they land on different answers; they are answering different implicit questions that the prompt failed to distinguish. A prompt asking for "a summary" of a technical document, for instance, leaves open whether the summary should be technical or accessible, and annotators will split along exactly that fault line.

Response style, verbosity, and complexity form a second class, and this one deserves attention because it is often mistaken for noise when it is actually a learnable pattern. Some annotators consistently reward longer, more elaborated answers; others consistently penalize them. That is a systematic bias with a direction, which means a reward model can, in principle, learn to detect and account for it, the same way it might learn any other feature of the data.

Value pluralism is the class that pluralistic alignment research exists to handle: genuine ethical or aesthetic disagreement across annotators, the kind that will not resolve no matter how well the prompt is specified or how well-trained the annotator pool is. Two annotators can read the same response to a question about a contested medical decision and split because they hold different values about autonomy versus caution. Collapsing that into a majority vote just picks a side and hides that a choice was made. It just picks a side and hides that a choice was made.

What collapsing disagreement into a majority label destroys

Three failure modes trace directly back to specific aggregation decisions, and each shows up differently in a deployed model.

Reward hacking and sycophancy come first. When annotators systematically favor fluent, confident-sounding answers over careful, hedged ones (a pattern that falls squarely in the response-style class above) the reward model learns that fluent certainty scores higher than careful accuracy. The policy then gets more willing to fabricate answers that sound clean, because sounding clean is what got rewarded during training, regardless of whether the underlying content held up.

Hallucination amplification follows a similar path. If confident, polished answers get consistently preferred during annotation, the reward model learns to reward confidence as a signal in itself, and the policy optimizes for the appearance of accuracy rather than accuracy. This is not a hypothetical risk; it is the direct, mechanical consequence of training a scalar predictor on labels that conflate fluency with correctness.

Representational bias is the third, and it is the one majority voting is structurally worst at handling. When a preference pair draws four annotators and three of them share a demographic or cultural background that shapes their reading of a response, majority voting doesn't cancel that skew, it amplifies it. The minority annotator's judgment doesn't get preserved as a data point about disagreement; it gets discarded as the outlier that lost the vote.

Random noise and systematic noise are not the same failure, and conflating them is part of the problem. Random noise mostly costs sample efficiency: the reward model needs more data to converge on the same signal. Systematic noise is worse, because it actively redirects optimization toward behavior humans never intended, and no amount of additional data fixes that; it just makes the model more confidently trained on the wrong target.

The paper "Hidden Consensus" (arXiv 2606.10569) names this collapse Preference-Validity Compression. The point is that when a task has multiple plural-valid responses (several answers that reasonable annotators would all accept) the aggregation step doesn't summarize the human preference behind them. It determines which of those acceptable responses remains visible to the optimizer at all, and the others simply disappear from the training signal as if they never existed.

That has a sharp implication for how teams read their own evaluation results. A model that looks like it has "degraded" on some later benchmark may not have degraded at all. It may have learned the preferences of a specific annotator subgroup faithfully, and that subgroup's preferences simply don't match whoever is running the evaluation. Calling that a regression, rather than a mismatch in whose preferences got encoded, is a diagnostic error with real consequences for what gets fixed and what gets left alone.

How annotator type and pool composition shape the signal before modeling begins

Who does the labeling changes what gets labeled. In one study comparing expert and external crowdworker annotators, consistency between the two groups came out to 70%, with the two groups landing on flatly opposite judgments 4.5% of the time.

Restrict that same comparison to samples with a clear preference (cases where responses weren't marginally different) and consistency climbs to 77%, while the opposite-judgment rate rises slightly to 6.5%. That pattern means the gap between experts and crowdworkers is driven disproportionately by the ambiguous cases, not the easy ones. Crowdworkers and experts mostly agree when the answer is obvious and diverge most in exactly the territory where careful judgment matters most.

Bring GPT-4 into the comparison as a third annotator and the picture gets more textured. Expert-versus-GPT-4 consistency is 64%, crowdworker-versus-GPT-4 is 44%, and in 9% of cases the expert group disagreed with both other groups simultaneously. That 9% is a genuinely distinct signal: it is not experts siding against a crowd, it is experts seeing something neither the crowd nor the model annotator saw.

The efficiency numbers underline why this is not just a fairness argument. Expert annotation reaches equivalent reward model quality using only 6 to 7% of the preference-pair volume that crowdsourced annotation requires. The old justification for cheap, high-volume crowdsourcing, that individual errors cancel out through aggregation, does not hold up here. Preference pairs are not the kind of data where averaging over noisy labels gets you closer to truth; the disagreement itself carries information that volume alone can't substitute for.

Technical approaches that treat disagreement as a first-class signal

The architectural fix follows directly from the diagnosis: move from a deterministic scalar reward model to a distributional one, a reward model that outputs a probability distribution p(r|x,y) rather than a single number. A distribution can encode annotator disagreement directly, as spread, rather than erasing it as a byproduct of averaging.

This reframing does more than add nuance. It opens the door to pessimism as a mitigation strategy: deliberately lowering the reward assigned in regions where the model is uncertain, which is a principled defense against reward hacking that a scalar reward model has no mechanism to support, because a scalar model has no concept of its own uncertainty to draw on in the first place.

Several concrete frameworks build on this idea. Mean-variance reward models track both a central estimate and a variance term, which lets the system distinguish a response with high consensus from one where annotators genuinely diverged, something a single averaged number can never represent. Ensemble-based reward models take a different route, training multiple reward models and combining them under one of three rules: mean aggregation, worst-case optimization, or uncertainty-weighted optimization.

Uncertainty-Aware Reward Modeling adds calibration through quantile-based conformal prediction, and it uses that calibrated uncertainty to treat uncertainty differently depending on where in the input space it appears, rather than applying one blanket correction everywhere.

Pairwise Calibrated Rewards for Pluralistic Alignment takes the most direct aim at value pluralism specifically. It learns a distribution over multiple reward functions, each one capable of inducing its own distinct aligned policy, and the criterion holding the whole thing together is pairwise calibration: for any two candidate responses, the share of reward functions in the ensemble that prefer one response should match the share of annotators who preferred it. The method learns this straight from pairwise preference data, and the research behind it found that even a small ensemble can represent a genuinely diverse preference distribution with reasonable fidelity.

What annotation platform design needs to look like when disagreement is signal

Standard annotation QA and RLHF preference collection want opposite things. A typical QA framework exists to resolve disagreement, to drive annotators toward a single, consistent label. RLHF workflows need the opposite: a way to preserve disagreement and structure it so it survives into the reward model rather than getting flattened before it ever gets there.

Tooling built for other domains doesn't transfer cleanly. Platforms designed around computer vision QA lean on IoU-based agreement scores, which work fine when there's a deterministic right answer, like a bounding box that either overlaps the ground truth or doesn't. Preference judgments about a chatbot response have no such deterministic target, so overlap-based scoring from computer vision QA applies poorly. Inter-annotator agreement statistics like Cohen's kappa, Fleiss' kappa, or Krippendorff's alpha are the more relevant measures for this kind of subjective task, and yet these are not always surfaced as first-class metrics inside annotation platforms, which is a meaningful gap given how central the number is to knowing whether the collected data means anything.

IAA measures catch random variance well, but they are blind to systematic bias, where every annotator in the pool gets pulled the same wrong direction. That situation produces a high agreement score and genuinely poor training data at the same time, because everyone agrees, and everyone is wrong together. Catching that requires comparing labels against some outside ground truth, not comparing annotators against each other, since annotator-to-annotator comparison is exactly the check that a shared systematic bias will always pass.

Building for this means a few concrete shifts in how annotation gets structured. Pairwise ranking templates should aggregate multiple human signals as training data in their own right, rather than compressing them into one winning label before the reward model ever sees the spread. Inter-annotator variance needs to be treated as structured data worth keeping. Consensus annotation workflows should retain the full distribution of responses collected. Annotator background metadata (whatever a platform is able to responsibly collect) should stay attached to each judgment, because Bayesian modeling approaches like NUTMEG depend on that metadata to separate ordinary competence noise from disagreement that is genuinely systematic. Calibration tasks, meaning known-answer items seeded quietly into an annotator's queue, give a way to benchmark individual annotators before they're trusted with high-stakes preference collection. And feedback to annotators needs to move past binary pass or fail, toward something structured enough that an annotator actually learns why a judgment was flagged, which is also the mechanism by which annotation guidelines get revised instead of staying frozen at version one.

Three contested models of what annotation is for

A disagreement that rarely gets named directly runs through all of this: what is an annotator actually doing when they click one response over another? "Three Models of RLHF Annotation: Extension, Evidence, and Authority" lays out three distinct answers, and the choice between them isn't cosmetic. It changes what aggregation method is even defensible.

The extension model assumes each annotator already holds a stable, pre-existing preference, and the annotation task's job is just to surface it faithfully. Under this model, disagreement is treated as measurement error almost by definition, and the natural response is diagnostic: build tools that test annotators for internal consistency and filter out anyone who looks like they're not applying a stable rule at all.

The evidence model rejects the premise that annotation is about preference in the first place. On this view, an annotator marking a response as "harmful" is offering evidence about a fact in the world, not reporting a personal taste. Disagreement here signals uncertainty about that fact, not a plurality of legitimate values, and that distinction matters enormously for aggregation: uncertainty about a fact should get resolved by gathering better evidence, not honored as diversity to be preserved.

The authority model breaks from both. Annotation, under this frame, is a normative act: annotators aren't reporting what they happen to prefer, they're being authorized to decide what the model should do. Disagreement here is a governance problem. No amount of better statistics resolves a genuine dispute about who gets to decide, and no distributional reward model, however well-calibrated, substitutes for an actual answer to that question.

Most current RLHF pipelines behave as though the extension model is simply true and the other two don't need discussing. The empirical record on disagreement suggests otherwise, and the technical work on distributional reward models is, in effect, an attempt to build infrastructure that doesn't require picking one model over the others in advance. Whether that's a genuine synthesis or a way of postponing a decision that eventually has to get made is, at this point, still an open question.