Section 13.3: Curriculum and automatic randomization

"Start with a world the policy can almost solve, then widen it each time it succeeds. Difficulty is a dial, not a fixed setting."

A Patient Curriculum Designer

What This Section Builds

OpenAI's engineers once spent weeks tuning a single randomization curriculum by hand, only to watch a slightly-too-wide setting turn a near-perfect Rubik's Cube policy into a hand that flailed: this section turns that fragile, intuition-driven dial into an automatic feedback controller that expands ranges only when the policy is stable enough to benefit, and shrinks or rebalances them when training produces only failures.

The goal is to record the difficulty schedule as part of the experiment, not as an invisible training convenience. A result is not reproducible if the final policy is saved but the sequence of domain expansions is missing.

Theory

A curriculum defines a time-indexed training distribution \(p_k(\theta)\), where \(k\) is the training phase and \(\theta\) contains randomized domain parameters. Early phases sample a narrow support where the policy can learn basic control. Later phases widen the support toward the deployment envelope. Figure 13.3A captures this dynamic at a glance: the randomization range expands when the policy succeeds and contracts when it fails, tracking a difficulty level just above the current skill horizon.

Automatic domain randomization closes the loop. The rolling success rate is the fraction of the most recent \(W\) episodes in which the policy reached its goal, recomputed after every phase as a moving measurement of how the policy is coping with the current range. If rolling success is above the target band, expand the domain. If it is below the band, hold or narrow it. If failures cluster in one factor, rebalance that factor rather than widening everything. In practice, this typically produces a large reduction in training episodes compared to a flat schedule, though the exact size of the reduction depends on the task and the tightness of the target band.

The compression effect matters in embodied AI because a real robot incurs physical wear on actuators and joints with every episode. Wasted training episodes under a flat randomization schedule translate directly into more sim-to-real cycles, more hardware exposure, and longer deployment timelines. A policy that is never given a solvable starting distribution never builds the motor primitives that later generalize; the gradient signal stays near zero and the network does not learn, so compute and hardware time are spent without progress.

The mechanism is gradient quality. When variation is too wide at the start, most sampled environments are beyond the current policy's ability to recover a reward signal, so the policy gradient is dominated by noise. Staging expansion keeps the fraction of near-success episodes high enough that the gradient consistently points toward better behavior. Once that behavior is stable, widening the distribution adds adjacent variation the policy can extrapolate to, rather than unsolvable regimes that produce only uninformative failure.

Worked Example

The following snippet shows a small automatic randomization update. It expands the friction range when rolling success is high, keeps it fixed inside the target band, and narrows it when the learner is failing too often.

# Update one curriculum range from rolling success.
# The goal is to keep training challenging without destroying exploration.
def update_friction_range(current_range, rolling_success):
    low, high = current_range
    if rolling_success > 0.80:
        return (max(0.10, low - 0.05), min(1.00, high + 0.05))
    if rolling_success < 0.45:
        center = (low + high) / 2
        return (center - 0.10, center + 0.10)
    return current_range

for success in [0.88, 0.63, 0.32]:
    print(success, update_friction_range((0.35, 0.65), success))

Practical Recipe

1. Choose a small set of curriculum-controlled factors, such as goal distance, friction span, clutter count, or lighting range, drawn from the visual, physics, sensor, and task parameter classes.
2. Define a target success band before training, for example 55 to 80 percent over the most recent evaluation window.
3. Expand only the factors whose current band is mastered, and rebalance factors that dominate failure labels.
4. Freeze a final transfer panel that the curriculum controller never sees.
5. Save the phase schedule, range updates, trigger metrics, and final policy checkpoint together.

The Curriculum As A Controlled System

Curriculum and automatic randomization become useful when the training distribution is treated as a controlled system. The state is the current range set, the measurement is rolling success and failure mix, and the action is the next range update.

The graduate-level habit is to separate three claims. The learning claim says the schedule lets the policy acquire behavior. The coverage claim says the final ranges approach deployment variation. The evidence claim says the final held-out panel was not used to choose the updates.

Keeping those three claims separate is a tooling problem as much as a conceptual one, since each claim needs its own logged trace, so the choice of simulator, config manager, and experiment tracker directly shapes how cleanly the curriculum can be reconstructed.

A robust implementation starts with a curriculum ledger that records the update rule, phase count, final range, and leakage guard in the same artifact as the evaluation metric. The steps below turn that ledger into a reproducible workflow.

1. Write a one-paragraph task contract with observation, action, success, and failure fields (for an ANYmal locomotion task: base velocity and joint encoders in, joint targets out, success = reaching the goal velocity without a fall, failure = base contact or foot slip).
2. Start with the smallest simulator config that exposes that contract faithfully, such as a single Isaac Lab environment before scaling to the 4,096 parallel clones ADR needs for stable rolling-success estimates.
3. Run one deterministic smoke test (fixed seed, frozen ranges) and one perturbation test (a single friction or payload step) before launching the full ADR loop.
4. Save a single result artifact containing the Hydra config, seed, rolling-success and failure-factor traces, rollout videos, and the policy checkpoint, all keyed to the curriculum phase index.
5. Compare ADR against flat randomization only when one LeRobot script evaluates both policies on the same held-out hardware panel.

Expected output: the printed trace should expose the update rule, final range, metric, and leakage guard. If one of those fields is missing, the example is not yet an evaluation artifact.

That same trace becomes the first place to look when results disappoint, because the logged schedule turns a vague training failure into a specific question about which curriculum decision went wrong. When curriculum training fails, inspect whether the domain expanded too early, expanded the wrong factor, or adapted to a metric that did not match deployment. Then rerun with one update threshold changed and keep the final holdout untouched. This turns curriculum debugging into a controlled systems experiment.

A policy trained without a curriculum ledger is a result without a method: the final numbers exist, but the sequence of decisions that produced them is gone.