Section 43.4: Dexterous RL with demonstrations

"Demonstrations teach the hand where the search should begin."

A Data-Hungry Dexterity Group

Give a 16-DOF robot hand a Rubik's cube and pure trial-and-error, and it will fumble for millions of steps before it accidentally stabilizes a single fingertip; feed it a few hundred human demonstrations first, and that same policy reaches stable contact in a few hundred rollouts, which is exactly why dexterous RL almost always starts from demonstrations or teleoperation data and then fine-tunes with reinforcement learning (the imitation stage is specifically behavior cloning, BC: supervised training of a policy to mimic recorded action sequences, used here to pretrain before RL fine-tuning) under domain randomization or privileged critics (auxiliary value networks trained with access to ground-truth simulator state that the deployed policy never sees, used only to sharpen the training signal). By the end of this section you will be able to implement the demo-to-RL curriculum switch yourself, using the same contact-entry and success-rate metrics the worked example measures.

It pulls together offline data, on-policy refinement, and contact-rich evaluation for tasks where random exploration would be too slow or too unsafe to be useful.

Theory

Figure 43.4.1 captures the loop this section builds: demonstration data bootstraps a policy, RL refines its contact behavior, and a verifier checks the result on real contact-rich tasks before the cycle repeats. Pure RL in high-dimensional dexterous action spaces often wastes experience before it discovers stable contact. Demonstrations move the policy into the right contact manifold, which makes policy-improvement gradients far more useful. In the Dactyl setup, random exploration needed on the order of 50,000 rollout episodes before the policy stumbled onto a stable fingertip contact. Seeding from a few hundred teleoperation demonstrations cut that figure to roughly 300 episodes, because the policy started inside the contact neighborhood rather than searching for it across the full 16-DOF space.

The hybrid pipeline therefore mixes imitation loss, value-based or policy-gradient updates, and heavy randomization. The system still needs an action interface and a recovery-aware reward or verifier to prevent reward hacking (the policy finding a way to increase the numeric reward without actually accomplishing the intended task).

Why domain randomization is not optional

Domain randomization matters here because a policy trained in a single simulated environment learns to exploit exact physical constants that differ on every real robot: joint friction varies between units, fingertip compliance shifts with temperature, and object surface texture changes with wear. Without randomization, the policy overfits to simulation physics and fails on transfer. By training across randomized friction, mass, and actuator delay, the policy must learn contact strategies that succeed under uncertainty rather than strategies tuned to one fixed set of parameters.

Mechanically, domain randomization samples a new physics configuration at the start of each rollout episode. The simulator draws friction coefficients, object mass, and sensor noise from specified ranges, then runs the episode with those values fixed. Over millions of rollouts, the policy gradient averages across this distribution, so the policy converges on behaviors that are robust across the range rather than optimal for any single configuration.

The objective below makes this two-stage structure explicit: a behavior-cloning loss \(\mathcal{L}_{BC}\) (weighted by \(\lambda_{BC}\)) initializes the policy on demonstrations, and the expected-return term then drives RL fine-tuning of the same parameters \(\theta\).

$$ \mathcal{L}(\theta)=\lambda_{BC}\,\mathcal{L}_{BC}(\theta)-\mathbb{E}_{\pi_\theta}\left[\sum_t r_t\right],\qquad \pi_{\theta_0}\leftarrow \text{BC on demos},\qquad \theta \leftarrow \text{RL fine-tuning} $$

Worked Example

Step 2 of that algorithm hinges on a concrete decision (when the policy has entered contact reliably enough to hand off to RL). In practice that decision is made with two measured quantities: bc_success, the full-task completion rate under pure behavior cloning, and contact_entry_rate, the fraction of rollouts that reach stable fingertip contact regardless of task completion.

The code below turns that switch into an explicit, measurable test on those two quantities.

# Decide when to switch from pure imitation to RL fine-tuning.
bc_success = 0.78
contact_entry_rate = 0.86
switch_to_rl = bc_success > 0.7 and contact_entry_rate > 0.8

phase = "rl_finetune" if switch_to_rl else "continue_bc"
print({"phase": phase, "bc_success": bc_success, "contact_entry_rate": contact_entry_rate})

Expected output: The expected result enters RL fine-tuning because the cloned policy already reaches stable contact often enough to make further exploration useful instead of random.

Consider a specific case: a 16-DOF hand learning to reorient a cube from 50 teleoperation demonstrations. After BC pretraining, the policy reaches the stable grasp contact state in 86% of rollouts but succeeds at the full reorientation in only 31% (it enters the right region but cannot recover from slip). After 2 million RL steps with domain randomization over cube mass (50 to 200 g) and finger friction (0.6 to 1.4), the full-task success rate rises to 74% while the contact-entry rate stays at 84%. The demonstrations provided the contact grammar; RL provided the recovery policy inside it.

Before reading on, guess: if a BC-pretrained dexterous policy already achieves 86% contact-entry rate, what fraction of that gain survives to full-task success before any RL fine-tuning?

Practical Recipe

1. Record demonstrations with enough variability to teach contact entry and small corrections.
2. Measure contact-entry rate explicitly before RL begins.
3. Use perturbations during RL that reflect realistic drop, slip, or pose errors.
4. Keep behavior-cloning and RL checkpoints for the same task panel so regressions stay visible.
5. Treat sim-to-real evaluation as part of the training plan, not as a last-day surprise.

One design principle ties the two phases into a single curriculum. The central insight for curriculum design is that the data question and the RL question are not separate: dexterous learning succeeds precisely because the initial demonstration data changes the effective exploration distribution before any policy gradient update runs.

Disturbance-aware evaluation is therefore essential. A dexterous policy that performs only on nominal states may have learned choreography rather than robust manipulation. A policy that succeeds only inside a simulator has not learned manipulation; it has learned a simulation, which is why sim-to-real transfer is typically treated as a required next step for a dexterous policy trained this way, since domain randomization narrows but does not eliminate the sim-to-real gap.

If RL regressions appear, ask whether the reward changed the contact style, the perturbations are unrealistic, or the demonstrations covered too narrow a contact manifold.