RLHF explained: from preference data to better model behaviour

What RLHF data represents

Reinforcement learning from human feedback uses human judgments to help a model distinguish more desirable outputs from less desirable ones. The judgments may be pairwise rankings, best-of-N selections, scalar scores or rubric-based labels.

The value does not come from preference alone. It comes from a preference task that clearly defines helpfulness, correctness, safety, tone, policy compliance or another target behaviour.

Rubrics reduce hidden inconsistency

Without a rubric, reviewers often optimize different things. One may prioritize brevity, another completeness, and another politeness. A strong rubric defines dimensions, precedence rules, examples and how to handle uncertainty.

• Define each scoring dimension independently.
• Include positive, negative and borderline examples.
• Specify when a safety or factuality failure overrides style.
• Allow abstention when evidence is insufficient.

Calibration creates a usable workforce

Reviewers should complete shared calibration tasks before production. Their decisions are compared against adjudicated examples and against one another. Disagreement is reviewed to refine the rubric or identify reviewers who need additional guidance.

For specialized domains, expert reviewers may be necessary because general preference cannot substitute for technical correctness.

Training and evaluation must stay distinct

A preference dataset used to optimize the model should not also serve as the sole benchmark for proving improvement. Holdout evaluation sets, adversarial cases and regression suites help prevent teams from mistaking familiarity with genuine behavioural progress.