Section 20.2: What transfers and what does not

"The skill transferred. The artifact it was tuned on did not. Knowing the difference is most of the job."

A Careful Control Loop

The reality gap from Section 20.1 does not close uniformly across a policy. Some parts cross it untouched while others shatter. Take the ANYmal quadruped, a four-legged robot developed by ETH Zurich and ANYbotics for industrial inspection, trained in Isaac Lab (NVIDIA's GPU-accelerated simulator that runs thousands of parallel robot environments at once) at ETH Zurich. The gait-phase encoder crosses to hardware intact. But the friction coefficient randomized over \([0.4, 1.0]\) in simulation, the 50 Hz control assumption, and the simulated joint-velocity range all need fresh hardware evidence. A useful transfer ledger names each of these explicitly: the randomized variables, the simulator assumptions, the on-robot measurement that confirms them, and any demonstration-learning handoff. It records them in one auditable artifact rather than a single sim-to-real verdict.

As Figure 20.2A illustrates, transfer is selective: high-level task structure may travel well, while sensing, timing, contact, and safety margins need fresh evidence on the robot.

That selectivity is exactly why a policy cannot be judged as a single block. An RL policy should be decomposed before deployment into representation, policy structure, value estimates, reward terms, low-level actuation, and evaluation metrics. These pieces do not deserve the same transfer claim because each one couples to different simulator assumptions.

When that coupling goes unaudited, the result is brutal: a policy that works in simulation but fails on hardware is not a policy; it is a very expensive proof that the simulator assumptions were wrong.

From verdict to triage

The key question is practical: which learned quantities can be reused, which must be calibrated, and which must be thrown away before the first hardware trial? This is called the transfer triage decision, and it determines whether the first hardware session takes two trials or two hundred. Figure 20.2B lays out this triage as a pipeline: policy components enter a ledger audit and are routed into one of four buckets before deployment.

Theory

Before assigning a bucket to any component, it helps to have a vocabulary for what a policy is made of. The transfer ledger and the four-bucket decision (reuse, recalibrate, constrain, discard) introduced above depend on this decomposition, so it is defined here first and used afterward.

Let a policy be decomposed as \(\pi(a_t\mid z_t)\), where \(z_t=f(o_{0:t})\) is the learned state representation. The representation \(f\) may transfer if it captures geometry, contact phase, or goal relation. The action distribution \(\pi\) may fail if the target robot has different torque limits, command latency, backlash, or compliance.

Reward models transfer even less automatically, in practice. Simulator rewards are typically dense, clean, and privileged; the real robot exposes only sparse events, noisy force estimates, delayed vision, and safety interlocks. A reward term that trained the policy well can be invalid as an evaluation metric on hardware.

Reuse holds when the learned quantity describes a topological or contact-phase invariant. "Approach until force exceeds threshold" survives a hardware swap because neither the geometry nor the contact event changes. Reuse fails when the quantity encodes a timing assumption (a fixed step frequency), a magnitude assumption (a specific torque budget), or a sensing assumption (a privileged depth channel absent on the real robot). The earlier these assumptions appear in training, the deeper they embed in downstream weights. A single miscalibrated normalization channel can require hundreds of hardware rollouts to diagnose. In one reported deployment case, teams needed roughly 400 to 600 real trials to re-converge when the mismatch surfaced on hardware, whereas flagging that same channel as recalibrate before deployment let the policy converge in under 20 trials, a roughly tenfold difference in hardware cost traced to one unaudited wrapper parameter. Actual numbers vary by task and hardware, but the direction of the effect is consistent: earlier detection of a miscalibrated channel costs far less than late discovery.

Why privileged state matters for real robots. A simulator's physics engine maintains a complete internal world state: exact joint angles, contact normals, object masses, and ground-truth pose. The policy trainer can read any of these directly. A physical robot has only sensor measurements, which are noisy, delayed, and partial. When a policy trains on privileged state, it builds dependencies on information that is absent at deployment. On hardware this produces silent failure: the policy receives zeros or stale values where it expects clean ground-truth inputs, and the resulting action distribution falls outside anything seen during training.

How the discard criterion works mechanically. During training, the observation vector \(o_t\) may include channels sourced from the simulator's internal state rather than from a sensor model. To identify these, compare each channel's source in the environment configuration against the robot's actual sensor list. Any channel that cannot be computed from onboard sensors alone is privileged. Remove it from \(\mathcal{O}_{\text{real}}\) before deployment. If the policy depends heavily on that channel, the reward term that required it must also be replaced by a hardware-measurable proxy before the first hardware trial.

Worked Example

Code Fragment 20.2.1 builds a small transfer ledger for a drawer-opening policy. The output distinguishes reusable task structure from components that need hardware calibration or safety constraints.

# Classify policy components before hardware deployment.
# The ledger prevents simulator-only conveniences from masquerading as transfer.
components = {
    "contact_phase_detector": "reuse",
    "camera_exposure_threshold": "recalibrate",
    "maximum_pull_force": "constrain",
    "privileged_hinge_angle_reward": "discard",
}

for name, decision in components.items():
    print(f"{name}: {decision}")

Expected output: each policy component has a deployment decision. A transfer plan that says "deploy the policy" without this inventory hides the most important engineering choices.

Practical Recipe

1. List the learned components: encoder, memory state, policy head, value head, reward terms, termination logic, and controller interface.
2. Classify each component as reuse, recalibrate, constrain, or discard.
3. Remove privileged simulator inputs from the deployed observation path.
4. Replace simulator rewards with hardware-measurable evaluation metrics.
5. Run a small hardware gate for every component marked reuse, especially if it touches force, contact, or delay.

Each learned quantity needs an explicit deployment status before the idea becomes useful. A policy checkpoint is not a single artifact from a transfer perspective. It contains representations, action distributions, value estimates, normalization statistics, and assumptions about reward and termination.

Because the checkpoint bundles all of those quantities together, the discipline is to pull them apart into distinct claims rather than a single verdict. The graduate-level habit is to separate four claims. The invariance claim says what remains true across sim and real. The calibration claim says what must be measured on hardware. The safety claim says what must be constrained before exploration. The evidence claim says which hardware gate proves the decision was justified.

A robust implementation starts with a transfer ledger and a hardware gate. The ledger records the deployment decision for each component. The gate records the smallest hardware test that can falsify that decision.

1. Create a component ledger before loading the trained policy on hardware.
2. Attach a validation gate to every reuse and recalibrate decision.
3. Prohibit privileged simulator channels from the hardware observation contract.
4. Save the ledger, gate result, trace, and failure label in one artifact.
5. Report transfer only for components whose evidence was collected under the same robot protocol.

When transfer fails, ask which ledger decision was wrong. A reuse failure means the task invariant was overestimated. A recalibration failure means the measurement protocol was too weak. A constrain failure means the safety envelope did not cover the robot's actual behavior. A discard failure means a simulator-only shortcut leaked into deployment.