Section 17.2: Learning to walk in minutes: the parallel-RL recipe

"The robot did not learn to walk faster. The simulator learned to fall faster, so the policy could recover faster, so the update arrived faster."

A Locomotion Researcher, Watching the Clock

Start the timer, pour a coffee, and by the time the cup is empty a quadruped that has never moved is trotting across randomized terrain: that is the literal claim behind the recipe Rudin et al. used to train ANYmal locomotion in under 20 minutes on a single GPU, with 4096 Proximal Policy Optimization (PPO) environments, a 24-step horizon at 50 Hz, terrain and command randomization, and RSL-RL as the runner. Reproducing that result on a Unitree Go2 or an ANYmal C means versioning four things together in one training artifact: the simulator contact model, the PPO rollout geometry, the reward terms, and the reset and terrain distribution. Change any one silently and the 20-minute wall-clock number stops meaning anything.

The parallel-RL recipe behind fast locomotion training turns on a few control decisions that make a huge rollout useful: short horizons, many environments, stable normalization, randomized starts, and a held-out evaluation panel. Figure 17.2A captures the discipline at the heart of the recipe: many robots practicing on varied terrain, while a separate test panel refuses to be impressed by training reward alone.

The key question is practical: how much simulated walking experience reaches the learner per update, and how do we keep that experience fresh enough that PPO is still optimizing the policy that collected it?

Theory

Suppose a locomotion run uses \(N=4096\) environments and a rollout horizon of \(T=24\) control steps at 50 Hz. One PPO update then contains \(98{,}304\) transitions, but each environment contributes only \(0.48\) seconds of fresh behavior before the policy updates.

This is typically the central tradeoff in practice, though its exact shape depends on the simulator's contact model and the task's reward horizon. Larger \(N\) increases batch size without lengthening policy lag. Larger \(T\) improves temporal credit assignment but lets the rollout drift farther from the policy that will be updated. Consider the alternative that shows why parallelism dominates. A single environment at the same horizon would need over 170 sequential hours of simulated walking to match the 1,966 aggregate robot-seconds that one parallel update delivers. The parallel version finishes that same work in under a second of wall-clock time. A single-environment baseline required 50,000 sequential episodes to converge; 4,096 environments running simultaneously needed only around 300 parallel updates. Each parallel update already contains the diversity that sequential training could accumulate only across hundreds of restarts.

Worked Example

Code Fragment 17.2.1 calculates the sample geometry for a typical fast locomotion run. The aggregate robot-seconds are large, but the per-environment horizon stays short to control policy lag.

# Size a parallel locomotion PPO update from first principles.
# Short per-environment rollouts keep policy lag small while N supplies scale.
num_envs = 4096
horizon = 24
control_hz = 50
minibatches = 8
ppo_epochs = 5

samples = num_envs * horizon
seconds_per_env = horizon / control_hz
aggregate_robot_seconds = samples / control_hz
minibatch_size = samples // minibatches
sample_reuse = ppo_epochs

print(f"samples per update: {samples:,}")
print(f"seconds per env before update: {seconds_per_env:.2f}")
print(f"aggregate robot-seconds: {aggregate_robot_seconds:,.1f}")
print(f"minibatch size: {minibatch_size:,}")
print(f"sample reuse per rollout: {sample_reuse} PPO epochs")

Expected output: the trace should make policy lag visible. If a recipe reports only total samples and hides horizon, minibatch size, and PPO epochs, it is hard to tell whether the update is fresh or over-reused.

Practical Recipe

1. Start with a short horizon such as 16 to 32 control steps, then increase only when the advantage estimate cannot span the gap between a stumble and its fall.
2. Use enough environments to keep minibatches large without reusing stale data too many times. If \(N\) is too small, minibatches shrink, gradient estimates become noisy, and the training curve stalls or oscillates even though the horizon and reward are correctly tuned; this is the practical symptom that tells you to raise \(N\) rather than \(T\).
3. Randomize terrain, commands, friction, mass, latency, and pushes by seed family, not by ad hoc global switches.
4. Track reward terms separately so a standing-still policy cannot hide behind a shaped reward total.
5. Evaluate without exploration noise on held-out seeds every fixed number of updates.

Being able to compute those quantities is the prerequisite; reading what they imply is the point. The recipe becomes useful when every speed claim is paired with a freshness claim: how old is the behavior the learner is updating against? A wide rollout with 10 PPO epochs may train quickly, but it also asks the learner to reuse behavior from an older policy. A narrower rollout with fewer epochs may use fresher data but underfill the GPU.

Report the whole recipe, not the headline time. A reproducible locomotion result names ten things: task randomization, reset curriculum (the schedule that decides which terrain difficulty or starting pose an environment resets into, so easy resets dominate early and hard ones are phased in as the policy improves), control frequency, horizon, action scaling, reward terms, policy architecture, normalization, evaluation seeds, and hardware. Omit any one and the wall-clock number stops being reproducible.

A robust implementation starts by freezing the recipe fields that affect learning speed. The example below records the batch geometry and policy-lag budget next to the evaluation panel, so a later table can compare runs without mixing different configs.

1. Lock the control frequency, horizon, and action decimation before tuning reward.
2. Log every reward term and reset reason, not only the total return.
3. Keep normalization statistics versioned with the checkpoint.
4. Save evaluation videos or state traces on held-out seeds at fixed update intervals.
5. Compare recipes only when one evaluation script computes success, fall rate, and command error in one pass.

# Store the recipe fields that make a fast locomotion run auditable.
# Policy lag is controlled by horizon and sample reuse, not by env count alone.
from dataclasses import dataclass, asdict

@dataclass
class LocomotionRecipe:
    envs: int
    horizon: int
    minibatches: int
    ppo_epochs: int
    control_hz: int
    eval_seed_panel: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

recipe = LocomotionRecipe(
    envs=4096,
    horizon=24,
    minibatches=8,
    ppo_epochs=5,
    control_hz=50,
    eval_seed_panel="terrain_v3_holdout_0000_0255",
)
print(recipe.as_row())

Recording those recipe fields makes runs comparable, but a comparable run is still only as trustworthy as what it does once it leaves the simulator. A policy that trains in minutes but falls on the first hardware step has not learned to walk: it has learned to survive the simulator. When the policy learns quickly and transfers poorly, inspect the reward terms and reset reasons before you change the network. Such a policy may minimize falls by exploiting termination, track commands only on easy terrain, or depend on privileged simulator signals that deployment will not provide.