Section 18.3: Goal-conditioned policies; hindsight experience replay

"Every failure is a success at a goal no one intended to set; hindsight makes that official."

A Goal-Conditioned Optimist

Consider OpenAI's Dactyl-style in-hand manipulation, where a Shadow Hand (a five-fingered robotic hand designed to match the degrees of freedom of a human hand) must rotate a block to an arbitrary commanded orientation. With sparse reward, a randomly initialized policy almost never reaches the requested pose. Nearly every rollout returns zero, so the gradient signal vanishes. Goal-conditioned policies plus hindsight relabeling turn that wall of failures into a usable training signal. Figure 18.3A previews the core idea: a trajectory that missed the requested goal is relabeled with the goal it actually reached, turning a failure into valid supervision.

This section develops the contract for policies of the form \(\pi(a \mid s,g)\), where \(g\) is a desired goal such as a target object pose, waypoint, door angle, drawer position, or language-grounded task state. The reward is also goal-indexed: \(r_g(s,a,s')\) asks whether the transition moved the world toward that particular goal.

For an embodied agent, goal conditioning is not optional convenience. A physical robot that hard-wires a single target into its weights requires retraining, reflashing, and re-validation every time the task changes. No real deployment can absorb that cost when goals shift daily. The goal vector also acts as a run-time safety lever: operators can command the same low-level controller to stop at a safe waypoint instead of the full target when a human enters the workspace, without touching the policy weights.

How the goal enters the network

Mechanically, the policy receives the goal \(g\) concatenated to or fused with the state \(s\) at every decision step. The critic then learns \(Q(s, a, g)\), so value estimates and gradients are always goal-specific. Changing \(g\) at inference time steers the same network toward a different attractor (the behavior pattern the policy converges to for a given goal, here the trajectory that maximizes goal-conditioned value) without retraining.

The key question is practical: how can a robot learn from an attempt that missed the requested goal but still reached a different, useful state? Figure 18.3B traces the answer end to end, showing how one failed episode produces both an original replay entry and a relabeled copy that feed the same policy update loop.

A robot that discards every failed attempt discards its own curriculum; hindsight experience replay is the refusal to waste what the robot already paid for in time and energy.

Theory

That librarian's filing system has a precise mathematical form, and writing it out shows exactly which fields HER touches and which it leaves fixed. A goal-conditioned replay buffer stores transitions as \((s_t, a_t, s_{t+1}, g, r_g)\). In sparse-goal tasks, the original reward may be zero for most failed trials. HER adds extra replay entries by replacing the desired goal \(g\) with a goal \(\tilde g\) that was achieved later in the same episode, then recomputing the reward \(r_{\tilde g}\). The practical effect is striking: on robotic manipulation tasks with sparse reward, a policy trained without relabeling typically needs around 50,000 rollout episodes to reach 50% success on the requested goal; the same policy with HER often reaches the same success rate in roughly 300 episodes on benchmark pushing and reaching tasks, a reduction of roughly two orders of magnitude in some reported settings (the exact ratio varies with task difficulty, goal space, and relabeling strategy, so treat 150-fold as an illustrative order of magnitude rather than a guaranteed multiplier) from changing nothing about the robot, the physics, or the reward function, only the labels on transitions already collected.

The mechanism works because the physical transition did happen. The same push, step, or grasp attempt can teach the value of moving from \(s_t\) to \(s_{t+1}\) under a different goal label. The evaluation, however, must remain on the original goal distribution, otherwise relabeling becomes a way to lower the task rather than learn it.

Worked Example

Suppose a block was supposed to end at position 10, but the robot pushed it to position 7. The original transition failed. HER can relabel part of the replay entry with goal 7, because that is what the trajectory actually achieved. Code Fragment 1 shows the relabeling step.

# Relabel failed transitions with goals the episode actually achieved.
# The original task remains unchanged for final evaluation.
episode = [
    {"state": 0, "action": "push", "achieved_goal": 3},
    {"state": 3, "action": "push", "achieved_goal": 7},
]
requested_goal = 10

for transition in episode:
    original_reward = int(transition["achieved_goal"] == requested_goal)
    relabeled_goal = transition["achieved_goal"]
    hindsight_reward = int(transition["achieved_goal"] == relabeled_goal)
    print(transition["action"], "g=", requested_goal, "r=", original_reward, "her_g=", relabeled_goal, "her_r=", hindsight_reward)

Expected output: the same physical transitions are visible under two labels: failed for goal 10, successful for goals 3 and 7. The relabeled reward is a replay trick, not a deployment metric.

The algorithm above takes the relabeling fraction \(k\) as an input; in practice \(k\) is chosen empirically, not derived. A common starting point is to relabel each real transition with 4 to 8 additional hindsight goals (the n_sampled_goal setting below), then compare requested-goal success across a small sweep, because relabeling too many goals per transition dilutes the minibatch with easy successes and relabeling too few wastes the free supervision the episode already produced.

Practical Recipe

1. Define the goal representation in task coordinates, not only in pixels.
2. Log both desired and achieved goals at every step.
3. Write a reward function that can be recomputed for any candidate goal.
4. Relabel replay entries with achieved goals from the same episode.
5. Evaluate only on held-out requested goals, with the original success threshold.

Once reward recomputation is trustworthy, the decision that most shapes how well HER generalizes is no longer how to relabel but what a goal even is. The main design choice is the goal space. Pose, image, language, and contact-state goals each create different generalization problems. Pose goals are easy to score but may miss semantic intent: a robot that places a cup at the correct \((x, y, z)\) coordinate scores full reward even if the cup is upside down. Image goals capture rich state but reward visual coincidence. Two images that look similar under \(L_2\) pixel distance (the straight-line, per-pixel numerical difference between two images, with no understanding of what the pixels depict) may map to very different physical configurations, such as a full cup versus an empty one in identical positions.

Language goals and contact-state goals sit at opposite ends of the precision-flexibility tradeoff. Language goals are flexible but need grounding: the instruction "open the cabinet" must map to a measurable joint angle or contact state before reward can be computed, and if that grounding is wrong the agent learns to satisfy the embedding, not the intent. Contact-state goals (gripper closed, surface touched, object grasped) are precise and verifiable but do not generalize across object geometries without a shared parameterization. Choose pose or contact-state goals when the task has a clear geometric success criterion; add language goals only when the goal distribution is too broad to enumerate in task coordinates.

The graduate-level habit is to enforce relabeling-evaluation separation, keeping the two distributions strictly apart. Relabeling changes the training distribution in the replay buffer. Evaluation samples desired goals from the task distribution and computes success without hindsight. Mixing these two distributions invalidates the comparison because the training trick becomes part of the reported task.

With those tools chosen, the order in which you assemble them matters as much as the tools themselves. A robust implementation starts by testing the reward recomputation function independently. If the function cannot score arbitrary achieved-goal and desired-goal pairs, the replay buffer cannot relabel transitions safely.

1. Choose a goal representation and success tolerance.
2. Verify reward recomputation on hand-written state-goal pairs.
3. Store desired goal, achieved goal, action, next achieved goal, and done flag.
4. Relabel a controlled fraction of replay samples with future achieved goals.
5. Report original-goal success, hindsight sample ratio, and goal-distribution coverage.

Code Fragment 2 captures the audit fields that make a HER run interpretable.

# Build one HER audit record for a goal-conditioned policy.
# The record separates replay relabeling from deployment evaluation.
from dataclasses import dataclass, asdict

@dataclass
class HERAudit:
    section: str
    goal_space: str
    relabel_strategy: str
    reward_recompute_test: str
    report_metrics: list[str]

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = HERAudit(
    section="18.3",
    goal_space="object xy pose in meters",
    relabel_strategy="future achieved goals from the same episode",
    reward_recompute_test="score three hand-written achieved/desired goal pairs",
    report_metrics=["requested_goal_success", "hindsight_ratio", "goal_coverage"],
)
print(record.as_row())

When HER fails, first check whether the achieved goals are too narrow, too noisy, or not physically meaningful, then whether the relabeled goals match states the robot can reproduce on purpose. Label each failure as one of four kinds: exploration, goal representation, reward-recompute bug, or evaluation-distribution mismatch.