Section 26.3: Skill discovery and hierarchical RL

"The most useful skills are the ones nobody thought to name in advance."

A Surprised Skill Designer

Why Hierarchy Matters

A quadruped that has spent millions of steps mastering one exact route across one exact floor will sprawl helplessly the instant a chair blocks its path, yet a robot that first carved that same experience into a handful of named behaviors (walk, turn, recover-from-fall) can replan around the chair before its servos cool. Hierarchy is what separates timing, contact, recovery, and sequencing so a high-level planner can select among those behaviors without pretending every low-level policy is deterministic.

Skill discovery asks whether useful behaviors can be learned before the final task reward is known: ETH Zürich's ANYmal quadruped, for instance, learns a repertoire of recovery and gait behaviors in Isaac Gym that it never explicitly requested for any single deployment. Hierarchical RL then decides how to select and compose those behaviors for a downstream objective, the way the affordance-scored skill library of language-as-a-high-level-controller lets a Google Everyday Robots mobile manipulator pick "find a sponge" then "wipe the spill" from hundreds of pretrained primitives rather than relearning torques for each kitchen.

A skill library without named, verifiable behaviors is not a library; it is a pile of trajectories waiting for a task that may never arrive. Figure 26.3.C below traces this pipeline from raw trajectories to a named skill library.

Consider a specific case. The DIAYN algorithm ("Diversity Is All You Need," Eysenbach et al., 2018, a method that discovers skills by rewarding the agent for reaching states that reveal which skill is running) trains a MuJoCo half-cheetah agent with no task reward at all. Over 2 million environment steps it discovers roughly 50 distinguishable locomotion behaviors, each labeled by a latent code \(z\). A downstream controller then trains for only 200,000 additional steps and selects among those pre-discovered behaviors to reach a target velocity. Without the skill library, a flat RL agent needs far more steps for the same target. In the DIAYN paper's velocity-tracking transfer experiment, the flat baseline typically needed roughly 50,000 downstream episodes to reach the performance the skill-conditioned agent matched in under 300. That is on the order of a 160x reduction in this particular experiment (results vary by task and seed), because the meta-controller searched over 50 named behaviors instead of the full joint-torque space. This two-phase structure is the core argument for hierarchical RL: hard exploration problems become easier when the search space shrinks to named skills, measured in dozens rather than millions of individual motor commands.

Formal Contract

If a skill is only useful once it can be named, verified, and selected, then the discipline lies in pinning down exactly what each of those guarantees means: that is what the option tuple formalizes.

Use the option tuple as an audit checklist: initiation states, internal policy, termination probability, and verifier must match the robot task. The Option-Critic architecture (Bacon et al., 2017), a method that learns the option's internal policy and termination function together by backpropagating through both, learns these fields end-to-end by differentiating through the termination function, while systems like SayCan (Ahn et al., 2022), a robot system that pairs a language model's skill suggestions with a learned feasibility score, ground the same tuple in a pre-built skill library where each entry carries an affordance score (a learned estimate of how likely a skill is to succeed from the current state) derived from a value function trained on real robot data. Both approaches expose the same checklist; they differ only in whether the fields are hand-specified or gradient-trained.

The combined skill-discovery and task objective makes this contract concrete. The expression below maximizes two terms at once: a diversity term that rewards skills for being distinguishable, plus the usual task return.

$$\max_{\pi,z}\; I(z;s_T) + \mathbb{E}\left[\sum_t r_{\mathrm{task}}(s_t,a_t)\right],$$

Reading the objective term by term

Here \(z\) is the latent skill code, the discrete or continuous label that identifies which skill is running. The terminal state the skill reaches is \(s_T\), and \(I(z;s_T)\) is the mutual information between them. This mutual information measures how reliably you can infer which skill ran just by looking at where it ended up. Maximizing \(I(z;s_T)\) pushes different skill codes toward visibly different outcomes, which is exactly the DIAYN diversity signal. The second term \(\mathbb{E}[\sum_t r_{\mathrm{task}}]\) adds the downstream task reward so the discovered skills also stay useful.

Map the option fields onto behavior trees, task graphs, finite-state machines, or task-and-motion planning nodes so start, act, stop, and verify remain inspectable. Figure 26.3.B shows this mapping end to end, from mission goal through the task graph to individual verified skills.

Worked Implementation

Code Fragment 1 for Skill discovery and hierarchical RL should expose initiation, progress, termination, verification, and failure reporting before connecting the skill to ROS 2, BehaviorTree.CPP, Drake, or a learned policy.

# Cluster short trajectory summaries into candidate skills.
# This toy discovery pass groups behaviors by displacement and contact evidence.
import numpy as np

summaries = np.array([
    [1.0, 0.0, 0.0],
    [0.9, 0.1, 0.0],
    [0.0, 0.0, 1.0],
    [0.1, 0.0, 0.9],
])
skill_names = []
for forward, lateral, contact in summaries:
    if contact > 0.5:
        skill_names.append("manipulate")
    elif forward > 0.5:
        skill_names.append("navigate")
    else:
        skill_names.append("unknown")
print(skill_names)

The expected output is only useful because each cluster now has a stable semantic label. A higher-level controller can select navigate or manipulate as reusable skills, whereas unlabeled embeddings would still be hard to schedule or verify.

Practical Recipe

Once the verified-execution loop above is in place, the remaining work is disciplined bookkeeping: the steps below turn that single-skill contract into a repeatable procedure for building and sequencing a whole library.

1. Name each skill with a verb and object: navigate_to_station, grasp_handle, dock_drone, or change_lane.
2. Write preconditions, postconditions, safety guards, timeout, and recovery behavior before training a policy.
3. Represent sequencing as a finite-state graph, behavior tree, or task-and-motion plan so failures have explicit routes.
4. Use language as a planner only after commands are grounded into a typed skill library with affordance checks.
5. Evaluate composition, not only individual success. Many failures occur when two correct skills meet at a bad boundary.

What actually goes wrong when a termination rule fires a half-second too late on a physical arm carrying a full coffee cup?

The termination condition deserves special attention because it determines when the high-level planner regains control. On a physical robot, a termination rule that fires too early hands a half-finished state to the next skill; one that fires too late holds actuators under load past safe limits, causing joint overheating or object slippage. These failures are silent in simulation because MuJoCo has no thermal model and contacts are assumed ideal.

Mechanically, \(\beta(s) \in [0,1]\) is a learned or hand-coded function giving the probability of ending the option at state \(s\). The policy samples it every timestep, and \(\beta(s) \approx 1\) releases control. Option-Critic trains \(\beta\) from downstream-return gradients, so states near a subgoal push it toward 1. Sensor-grounded hard overrides on force, time, and pose error keep \(\beta\) from staying near 0 when the policy is stuck.