Section 18.5: Human preferences and learned reward models (RLHF for control)

"Human preferences are the reward signal. All the usual problems now apply to a model of a person."

A Careful Control Loop

Picture a policy that has just earned its highest reward score ever. The gripper it controls leaves a fresh scratch across the table on every rollout. A learned preference reward reports success anyway, unless the builder exposes each force budget, clearance constraint, exploration bonus, and deployment risk as a separately auditable term. Bury those terms in one scalar return, and a MuJoCo or Isaac Lab rollout can quietly inflate them.

This section develops the contract for reward models in embodied control. Figure 18.5A previews the whole loop: a human compares two short trajectory clips, that preference trains a reward model, and the reward model drives PPO fine-tuning of the policy. A learned reward model maps a trajectory segment \(\tau\) to a scalar score \(\hat R(\tau)\). The training data is not a numeric reward label for every state. It is usually a set of comparisons such as "clip A is safer, smoother, or more successful than clip B." In practice, a hand-written reward for a warehouse sorting task may require weeks of parameter tuning across hundreds of rollout hours, while preference labels from 500 clip comparisons can match or surpass that performance in a single overnight training run (Christiano et al., 2017; Lee et al., 2024).

The key question is practical: when should the builder trust learned preferences more than a hand-written reward, and how can the builder detect when the policy starts exploiting the reward model? Figure 18.5B traces the three-stage pipeline that answers this: collect comparisons, train the reward model, and optimize the policy under a KL penalty, where KL (Kullback-Leibler) divergence measures how far the updated policy's action distribution has drifted from the reference policy, and penalizing it keeps updates close to familiar behavior, with independent physical metrics auditing the model against exploitation.

Theory

A common preference model uses a Bradley-Terry form, where the probability that one item is judged better than another is a logistic function of the difference between their underlying scores. If a rater prefers trajectory \(\tau_A\) over \(\tau_B\), the model predicts

$$P(\tau_A \succ \tau_B)=\frac{\exp(\hat R(\tau_A))}{\exp(\hat R(\tau_A))+\exp(\hat R(\tau_B))}.$$

Training minimizes the negative log probability of the observed preference. In embodied control, the rater prompt must be precise: should the rater prioritize task completion, smoothness, speed, clearance from humans, object damage, or recovery behavior? A vague prompt trains a vague reward model.

Why ordinal judgment fits physical control

Why this matters for physical robots. A robot acting in the world cannot roll back a mistake: a knocked object stays fallen, a bruised joint accumulates wear, a startled co-worker loses trust. The Bradley-Terry form captures ordinal human judgment rather than asking raters to assign precise numeric scores, which is unreliable for physical behaviors where magnitude is hard to judge but relative preference is clear. This makes it the right tool when the cost of a bad reward is irreversible physical harm rather than a recoverable software error. A reward model that a policy can fool is not a reward model; it is a scoreboard waiting to be gamed.

How it works in practice. A neural network takes a trajectory segment (states, actions, optionally sensor readings) as input and outputs a scalar score. For each labeled pair, the network scores both clips, and the Bradley-Terry formula converts the score gap into a predicted preference probability. Backpropagation through the negative log-likelihood loss adjusts the weights, so clips the raters preferred reliably receive higher scores. After enough labeled pairs the network generalizes. It scores new clips it has never seen, providing a reward signal the PPO loop then maximizes. Consider the leverage. In the benchmark tasks reported by Christiano et al. (2017), learning a comparably expressive reward through random exploration alone demanded on the order of tens of thousands of rollout episodes, while roughly 500 carefully chosen human comparisons produced comparable signal quality in a single overnight run; the exact ratio depends heavily on the task and should not be assumed to transfer unchanged to a new embodiment. Each labeled pair condenses substantially more environmental experience than the agent could stumble onto by itself.

Worked Example

Suppose raters prefer a slower grasp that avoids scraping the table over a faster grasp that succeeds but collides. Code Fragment 1 computes the probability assigned to the preferred clip and the corresponding loss.

# Compute one preference-model loss for two trajectory clips.
# The preferred clip should receive the higher learned reward score.
from math import exp, log

reward_safe = 1.4
reward_fast_collision = 0.2
prob_safe_preferred = exp(reward_safe) / (exp(reward_safe) + exp(reward_fast_collision))
loss = -log(prob_safe_preferred)

print("P(safe preferred)=", round(prob_safe_preferred, 3))
print("preference_loss=", round(loss, 3))

Expected output: the probability should rise when the preferred clip receives a higher score. If the score gap grows on training data but fails on held-out clips, the reward model has memorized the comparison set rather than learned the rater's criterion.

Practical Recipe

1. Define the rater rubric before collecting comparisons.
2. Sample diverse trajectory pairs, including failures, recoveries, near misses, and easy successes.
3. Track rater disagreement and remove or investigate ambiguous pairs.
4. Validate the reward model on held-out clips and adversarial high-score rollouts.
5. Report policy performance using task success and safety costs, not reward-model score alone.

Learned rewards are most useful when the desired behavior is hard to write as a formula but easy for trained raters to compare. Smooth recovery, respectful distance, gentle contact, and task style often fit this pattern. They are least useful when the rater cannot observe the relevant evidence, such as hidden force, internal wear, or a delayed safety consequence.

Treat reward-model optimization as a distribution shift. The model learns from comparison clips; the policy then searches for actions that maximize it, drifting away from that training distribution. So high reward-model scores should trigger adversarial review, uncertainty checks, and independent embodied metrics.

Choosing those tools well only pays off if the surrounding record is disciplined, because the tool captures the scores but not the human process that gave them meaning.

A robust implementation records the human-labeling process as carefully as the RL run. Without the rater protocol and comparison distribution, a learned reward score is not interpretable.

1. Write a rater rubric with ranked criteria and tie rules.
2. Collect comparison pairs across easy, hard, unsafe, and ambiguous cases.
3. Train the reward model and report held-out pair accuracy plus calibration.
4. Optimize the policy under the reward model while saving high-score rollouts.
5. Evaluate with independent success, safety, and intervention metrics.

Code Fragment 2 records the minimum audit fields for a learned reward used in control.

# Build one preference-reward audit record for control.
# The fields connect human labels, model validation, and policy evaluation.
from dataclasses import dataclass, asdict

@dataclass
class PreferenceRewardAudit:
    section: str
    rater_rubric: str
    validation_check: str
    exploitation_probe: str
    deployment_metrics: list[str]

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = PreferenceRewardAudit(
    section="18.5",
    rater_rubric="prefer task success, then gentle contact, then smooth recovery",
    validation_check="held-out pair accuracy plus disagreement review",
    exploitation_probe="top reward-model rollouts inspected for hidden contact",
    deployment_metrics=["task_success", "contact_cost", "human_intervention_rate"],
)
print(record.as_row())

That audit record is also what makes failures diagnosable rather than mysterious: once the rater protocol, probes, and metrics are logged, a failure can be traced to a specific stage instead of guessed at. When a learned reward policy fails, assign the failure to rater ambiguity, dataset coverage, reward-model overfitting, optimization exploit, or missing physical evidence. Then add comparison pairs that target that failure and rerun the policy evaluation on the same seed panel.