Section 20.4: Fine-tuning on hardware; safe real-world RL

"Simulation was a forgiving teacher. Hardware charges for every failed exploration in torque, time, and sometimes metal."

A Robot That Learned the Hard Way

The first bad gradient on real concrete does not show up as a number in a log file. It shows up as a Unitree A1 slamming a knee joint past its torque limit, and there is no undo button. So before the robot takes a single gradient step, the team should pin down four things in one transfer ledger. First, the randomized variables (friction range, payload mass, motor strength in Isaac Lab). Second, the simulator assumptions baked into the contact model. Third, the real-world measurements that set each bound (a force-torque sensor reading, a measured 38 ms actuator delay). Fourth, the handoff point where logged demonstrations or proprioceptive history take over from the sim-trained base. ETH Zurich's ANYmal locomotion work and Berkeley's A1 deployments both treat this bookkeeping as a precondition for touching hardware, not an afterthought. The "safety gates" mentioned here are defined formally, as explicit inequality constraints, in the Theory section below; for now, treat a gate as any check that can block a proposed action before it reaches the robot.

Why the ledger comes first

A transfer ledger matters on real hardware because physical consequences are irreversible. If randomization bounds, contact parameters, or sensor noise levels are unrecorded, a safety gate that passed in simulation may silently fail on the robot; an engineer then cannot distinguish a bad update from a bad assumption. Unrecorded handoff points also make it impossible to audit whether a reward gain came from better hardware adaptation or from a simulator assumption that happens to match the specific test surface but will not generalize.

The ledger pairs each sim parameter (friction range, mass uncertainty, delay model) with the real measurement or worst-case bound behind it, and flags every assumption never validated on hardware. During fine-tuning it doubles as a change log: widen a gate threshold or hit a new terrain, and you append the responsible parameter and its measurement. The result is a traceable chain from simulator choices through safety decisions to hardware outcomes.

A hardware fine-tuning protocol has five parts: a frozen simulator-trained policy, a real-robot evaluation gate, a constrained update rule, a human or automated stop channel, and a rollback checkpoint. As Figure 20.4A shows, hardware fine-tuning earns its extra performance only when every proposed action passes through explicit safety gates before reaching the robot. The point is to improve transfer without turning the robot into an uncontrolled training environment.

Figure 20.4B traces this constrained loop end to end: the frozen base and trainable residual combine into a proposed action, the safety gate filters it, and gradient updates flow back only to the residual adapter. The key question is practical: what evidence lets the team safely move from evaluation-only rollouts to limited policy updates on hardware?

A policy that works in simulation but fails on hardware is not a policy; it is an aspiration that has never met friction.

Theory

To answer that question of when updates are safe, the gated loop sketched in Figure 20.4B has to be made precise, which means writing the safety filter and the update rule as explicit mathematical objects.

A hardware fine-tuning loop is written as a constrained update: choose action \(a_t \sim \pi_\phi(a_t\mid o_t)\) only if \(g_i(o_t,a_t) \leq 0\) for every safety gate \(i\). The gates may encode workspace limits, force limits, speed limits, joint-temperature limits, distance from humans, controller health, and intervention triggers.

The update rule should be smaller than the safety case. A common pattern is to freeze the perception stack and low-level stabilizer, then adapt a residual policy, gain schedule, or high-level action bias; this is called the frozen-base, small-adapter principle, and it limits the blast radius of any single bad gradient step on hardware. That limits the blast radius of learning while still allowing hardware-specific improvement.

Consider a specific case. Kumar et al. (2021) deployed RMA (Rapid Motor Adaptation) on an A1 quadruped. They froze the base locomotion policy, trained in Isaac Gym on thousands of parallel environments, and adapted only a small encoder that reads 50 ms of proprioceptive history (the robot's own joint-position, joint-velocity, and orientation readings, as opposed to external sensing like vision). The adaptation module updated online from roughly 200 real steps per terrain type, whereas the base policy had required millions of sim steps to reach the same terrain. The frozen base did the heavy lifting, so the adapter could learn a new terrain in about the number of steps a child takes crossing a parking lot. Joint torque stayed capped at 33 Nm and episode length at 20 seconds per trial. The A1 recovered from terrain mismatches that the frozen base policy failed on, and neither the low-level PD controller (proportional-derivative controller, the fast feedback loop that converts a target joint position into a motor command) nor the reward shaping changed. This two-layer structure (frozen base, small trainable adapter) directly implements the constrained update pattern described here.

Worked Example

The numeric trace above showed the gate accepting and rejecting individual proposals; the same logic reduces to a few lines of code that any hardware loop can call before an action leaves the controller.

Code Fragment 20.4.1 implements an action gate for a residual pushing policy in four checks. The proposed residual is allowed only if speed, force, actuator delay, and workspace checks stay within the approved envelope.

# Gate a residual action before it reaches the real robot.
# A hardware fine-tuning loop updates only after safety checks pass.
proposal = {"speed_mps": 0.18, "force_n": 14.0, "delay_ms": 38, "workspace_ok": True}
limits = {"speed_mps": 0.20, "force_n": 15.0, "delay_ms": 45}

passes_gate = (
    proposal["workspace_ok"]
    and proposal["speed_mps"] <= limits["speed_mps"]
    and proposal["force_n"] <= limits["force_n"]
    and proposal["delay_ms"] <= limits["delay_ms"]
)

print(f"passes_safety_gate={passes_gate}")

Expected output: a hardware fine-tuning diagnostic reports whether the action passed the gate and which constraint would have blocked it. If blocked actions are not logged, the learner's data distribution cannot be audited.

Practical Recipe

1. Begin with evaluation-only hardware rollouts from the frozen sim-trained policy.
2. Define safety gates for workspace, speed, force, joint limits, temperature, actuator delay, human proximity, and controller health.
3. Choose the smallest trainable subset: residual action, gain schedule, adapter, or high-level command bias.
4. Set an intervention budget and rollback rule before the first update.
5. Report learning progress together with safety events, blocked actions, human interventions, and hardware resets.

The idea in this section becomes useful when the hardware protocol is explicit enough to audit. The protocol names the frozen policy checkpoint, the trainable residual, the gate conditions, the monitoring frequency, the intervention policy, and the rollback checkpoint. Without these details, hardware fine-tuning becomes an anecdote.

The graduate-level habit is to separate three claims. The improvement claim says the policy performs better. The safety claim says exploration stayed within the approved envelope. The deployment claim says the final policy remains robust under held-out initial conditions and delay tests.

A robust implementation starts with a safety gate document and a logging schema. The gate document states what can stop an action. The schema records what was proposed, what was blocked, what was executed, and whether the policy was updated afterward.

1. Run the frozen policy under the same gates used for fine-tuning.
2. Train only after the frozen-policy intervention rate is below the predeclared threshold.
3. Update the smallest residual or adapter that can plausibly fix the observed gap.
4. Checkpoint before every update batch and define automatic rollback triggers.
5. Evaluate the final policy on held-out starts, delay perturbations, and safety-envelope edges.

When hardware fine-tuning fails, assign the failure to one of four categories: unsafe proposal, wrong gate, harmful update, or misleading reward. Then rerun the frozen checkpoint against the same starts to decide whether the update caused the failure or merely exposed a preexisting transfer gap.