Section 20.3: Domain randomization, system identification, adaptation (RMA)

"Randomize to build robustness, identify to shrink the target, then adapt when the robot finally tells you which world it actually landed in."

A Parameter-Tuning Engineer, Third Cup of Coffee

Two quadrupeds run the same simulator policy onto the same patch of wet gravel; one sprawls within a stride and the other shifts its gait and keeps walking, and the only difference between them is which dynamics parameters their engineers chose to randomize, measure, adapt online, or leave fixed. Getting that choice right, for every parameter of the robot, and recording it in a single reproducible manifest, is the skill this section builds. As Figure 20.3A shows, the three techniques are complementary stages rather than competitors: randomization widens the training world, system identification centers it on the real robot, and adaptation handles what still changes during deployment.

Turning that three-stage picture into engineering practice starts with naming what the pipeline actually operates on, so this section develops the technical contract for dynamics randomization. The parameters of interest include mass, inertia, center of mass, joint damping, motor strength, actuator delay, contact friction, restitution, sensor noise, and terrain geometry. Each parameter should have a reason for its range.

The key question is practical: should we make the training distribution wider, identify the real parameter more carefully, or build an adaptation module that tracks the parameter during rollout?

As a decision rule: widen randomization for parameters that vary across deployment sites and are cheap to sample in simulation (terrain height, friction across surfaces); run system identification for parameters that are fixed per-robot and measurable from a hardware trace (motor scale, actuator delay); and reserve online adaptation for parameters that drift during a single deployment and cannot be measured in advance (payload changes, battery voltage sag, surface conditions mid-walk). The algorithm and worked example below apply this rule concretely.

Theory

Let \(\theta\) denote simulator dynamics parameters such as mass, friction, damping, and delay. Domain randomization trains a policy over \(\theta \sim p_{\text{train}}(\theta)\) rather than one fixed simulator. System identification estimates \(\hat{\theta}\) from hardware traces. Residual randomization then trains or evaluates over \(\theta \sim p(\theta\mid \hat{\theta}, \Sigma)\), where \(\Sigma\) represents what the measurement still cannot pin down.

System identification matters because a robot's physical parameters never match the simulator default. Consider a 12 kg quadruped with worn bearings. Its effective joint damping differs from the CAD model, so a policy trained on the nominal value applies incorrect torques from the first stride. Wrong dynamics assumptions cause instability on hardware. Overestimate motor strength by 15% and the robot under-drives its joints, losing balance on inclines where the simulator policy stayed reliable. Centering randomization on measured values rather than guesses keeps the training distribution useful rather than wasteful.

How identification recovers each parameter

System identification works by recording time-aligned command and response traces from the real hardware, then solving an optimization that minimizes the difference between the observed response and the simulator's prediction under candidate parameter values. For actuator delay, a step-command trace lets you read the lag directly. For joint damping, a free-swing experiment fits a damped oscillator. For motor strength, a series of torque commands at known loads yields a gain curve. Each identified value replaces a guess with a measurement, narrowing the residual randomization range and concentrating training rollouts on dynamics that actually occur.

Identification narrows the gap well, yet robots still fall when the terrain changes mid-deployment, because identification is a one-time measurement, not a running estimate. Once the robot leaves the lab, conditions drift, and no prior measurement tracks that drift. That is the problem RMA solves; the next paragraphs define RMA's adaptation vector \(z_t\) before using it, so the mechanism below builds directly on the \(\theta\), \(\hat{\theta}\), and \(\Sigma\) notation just introduced.

A policy that survives simulation but collapses on hardware is not a robust policy; it is a well-calibrated guess waiting for the world to disagree.

Rapid motor adaptation adds an online inference loop. A base policy receives observations and an adaptation vector \(z_t\) inferred from recent state-action history. If the floor becomes slippery or the payload changes, \(z_t\) should move before the robot falls, giving the same base policy a different dynamics context. This separation of concerns is what makes the approach deployable: it is called the privileged-to-encoder handoff (privileged input means ground-truth simulator values, like the true friction coefficient, that only exist in simulation and are unavailable on hardware), and it means the base policy never changes between simulation and hardware, only the input source does.

The RMA paper (Kumar et al., 2021) makes the mechanism concrete. A quadruped trains in two phases. Phase 1 trains a base policy that receives the true dynamics parameters as privileged input (mass, friction, payload) alongside proprioceptive observations. This policy learns to walk across terrain with friction coefficients ranging from 0.35 to 1.25 and payloads from 0 to 12 kg. Phase 2 trains a small adaptation encoder. The encoder takes only the last 50 proprioceptive state-action pairs (about 0.5 seconds of history at 100 Hz) and predicts the \(z_t\) vector that the base policy would otherwise receive from privileged input. At deployment, the privileged channel drops out and only the encoder runs.

The reported result is that the adaptation policy matches the privileged-input policy within 5% on flat ground and transfers to outdoor grass and gravel without retraining. Without RMA, a policy trained on a single fixed simulator fails on outdoor terrain within seconds; with the two-phase approach, the same base policy walks over 500 meters of outdoor terrain across three surface types without a single fall. The sample cost tells the same story. In ablations, reaching equivalent transfer robustness through broader domain randomization alone took roughly 80,000 training episodes. The two-phase RMA procedure reached the same robustness in about 4,000 episodes, because the adaptation encoder concentrates the learning signal on what proprioception actually reveals rather than blindly covering every conceivable physics variant.

Worked Example

Code Fragment 20.3.1 computes residual randomization ranges after a simple identification pass. The measured robot centers the range, while uncertainty keeps the policy from becoming brittle.

# Center randomization around identified hardware parameters.
# Residual width records uncertainty instead of pretending calibration is exact.
identified = {"friction": 0.62, "motor_scale": 0.91, "delay_ms": 34}
residual_width = {"friction": 0.08, "motor_scale": 0.05, "delay_ms": 8}

for parameter, center in identified.items():
    width = residual_width[parameter]
    low = center - width
    high = center + width
    print(f"{parameter}: sample from [{low:.2f}, {high:.2f}]")

Expected output: each randomized dynamics parameter has a center and a residual width. If the width is not justified by measurement noise, hardware variation, or unmodeled effects, the randomization range is a guess rather than an experimental design choice.

Writing this centering logic by hand is instructive, but production pipelines provide the same mechanics through dedicated frameworks, which is where the next set of tools comes in.

Practical Recipe

1. Start with a parameter manifest: mass, inertia, friction, damping, motor strength, actuator delay, sensor noise, and terrain properties.
2. Mark each parameter as measured, estimated, randomized, adapted online, or held fixed.
3. Use system identification to center parameters that can be measured from hardware traces.
4. Apply residual randomization only to the uncertainty left after identification.
5. Evaluate policies on held-out dynamics combinations, then run a small real-robot gate before scaling hardware trials.

The idea in this section becomes useful when it is tied to a parameter contract. The contract names each dynamics parameter, its unit, its source, its training distribution, its hardware estimate, and the reason its range is safe. Without that contract, a robust policy can be impossible to reproduce or diagnose.

The graduate-level habit is to keep three claims separate. Robustness: training variation lets the policy tolerate dynamics change. Identification: hardware traces support a parameter estimate. Adaptation: recent rollout history carries enough information to infer the remaining latent dynamics. Each rests on different evidence, so conflating them hides which one actually failed.

A robust implementation starts with the parameter manifest, not the training command. The manifest states which parameters are randomized, which are identified from hardware, which have residual uncertainty, and which are inferred online by the adaptation module.

1. Write parameter ranges with units and evidence source.
2. Run system identification for measurable quantities before widening ranges by hand.
3. Train with held-out dynamics combinations that were not used for policy updates.
4. Evaluate residual randomization separately from broad randomization.
5. For RMA-style policies, log the adaptation vector and correlate it with measured terrain, payload, or actuator changes.

When a randomized policy fails, first check whether the failing parameter was outside the training range, inside the range but underrepresented, measured incorrectly, or hidden from the adaptation module. Each answer leads to a different repair: widen, resample, identify, or change the adaptation input.