Section 35.2: Cross-embodiment training and transfer

"A trajectory is not transferable until every hidden unit, frame, and control rate has been made painfully explicit."

An Embodiment Auditor

The Real Problem Is Not Data Volume, It Is Data Meaning

A warehouse arm and a humanoid can both grasp the same cup and log the same success, yet still poison each other's training. One says "move 5" and means centimeters; the other means meters; nothing in the raw data reveals the difference. Pooling demonstrations from many robots only helps if the shared learner sees comparable events. As Figure 35.2A illustrates, several robot bodies contribute to one policy only after their trajectories are translated into a shared contract, not simply mixed as if every robot meant the same thing. A mixed dataset can silently combine different camera frames, control frequencies, action saturations, and success definitions. When that happens, the model spends capacity learning translation noise instead of task structure. In the Open X-Embodiment experiments (2023), a policy trained on raw-mixed data from 22 robots transferred to a held-out robot at roughly 30% success. After unit normalization and metadata conditioning, the same architecture reached 68% on the same tasks, with no change in model size or training compute. Doubling the raw dataset size without fixing the schema moved success from 30% to 34%: two months of additional data collection bought less than proper unit normalization did in an afternoon of engineering.

This is why cross-embodiment training lives at the boundary between datasets and policy design. The data schema is part of the model architecture. If the schema is wrong, added capacity only lets bigger models memorize the wrong equivalences more efficiently.

Before going further, one term needs pinning down: "canonical" here just means a single shared convention (one unit system, one gripper scale, one clock) that every robot's raw data is converted into before the policy ever sees it, so the policy always trains on the same kind of number regardless of which robot produced it.

A Canonical Latent Interface

A common pattern, shown in Figure 35.2B below, is to map robot-specific observations and actions into a canonical latent contract:

$$z_t = E_\text{obs}(o_t, m_r), \qquad a_t^{\text{can}} = N_r(a_t), \qquad \hat a_t = A_r^{-1}(\pi_\theta(z_t, q_t)).$$

Here \(m_r\) is embodiment metadata, \(N_r\) normalizes robot-specific actions into a canonical representation, and \(A_r^{-1}\) maps the canonical action back into robot-specific commands. The policy \(\pi_\theta\) only works if those adapters preserve the semantics that matter for transfer.

Code Fragment 1 shows the smallest useful version of that normalization idea, where a canonical range is a shared numerical scale (here, a common position unit and a 0-to-1 gripper value) that every robot's raw commands are converted into before training. The point is not numerical sophistication. The point is to make the unit conversion visible.

# Normalize position and gripper commands from two robots into one canonical range.
# The policy can only share data when these conventions are explicit.
robots = {
    "arm_a": {"xyz_scale_cm": 1.0, "gripper_closed": 0.0},
    "arm_b": {"xyz_scale_cm": 2.5, "gripper_closed": -1.0},
}

def normalize_command(robot_name: str, dx_cm: float, gripper_value: float) -> tuple[float, float]:
    meta = robots[robot_name]
    canonical_dx = dx_cm / meta["xyz_scale_cm"]
    canonical_gripper = 1.0 if gripper_value == meta["gripper_closed"] else 0.0
    return canonical_dx, canonical_gripper

print(normalize_command("arm_a", dx_cm=1.0, gripper_value=0.0))
print(normalize_command("arm_b", dx_cm=2.5, gripper_value=-1.0))

(1.0, 1.0)
(1.0, 1.0)

The expected output is matching canonical commands for two different robots after embodiment-specific decoding and normalization. If these tuples diverged for the same semantic command, pooled training would quietly mix incompatible actions and poison cross-embodiment transfer.

Before reading on, consider this: if two robots perform the exact same cup-grasp but one sends position in centimeters and the other in meters, how many training steps does the shared backbone waste learning that unit conversion before it can learn anything about grasping?

Why The Shared Latent Space Enables Transfer

Why does this latent interface enable sharing at all? The encoder \(E_\text{obs}\) maps the same physical event to nearby points in \(z\)-space regardless of which robot produced it. Consider a Franka Panda gripper closing at 0.08 m/s on a cup and a WidowX 250s gripper closing at 0.04 m/s on the same cup: both map to nearby \(z\) coordinates. That proximity lets gradient updates from the WidowX slice improve the Franka policy, and vice versa. In Open X-Embodiment, a ViT-B/16 encoder (a Vision Transformer image encoder with roughly 86 million parameters) conditioned on embodiment metadata reached same-task cluster purity above 0.75 across 22 robots, where cluster purity is the fraction of trajectories in a cluster that share the same task label. Without metadata conditioning, purity dropped to 0.41: the backbone spent its capacity distinguishing robots rather than learning tasks.

A poorly designed canonical space places trajectories from different robots in unrelated clusters. The shared backbone then learns embodiment identity instead of task structure. This failure has a name: the same-task clustering test. The test asks: "Do same-task, different-embodiment trajectories cluster together in \(z\)-space?" If they do not, the interface is the wrong abstraction, and transfer stays near zero even with a large dataset. The test also matters for hardware safety. A robot with poor cluster overlap receives gradients that push its representations toward unrelated motions. On physical hardware, a mis-timed command can damage actuators or drop a payload. The clustering succeeds when the encoder trains with a contrastive or task-conditioned objective. That objective penalizes distance between same-task pairs regardless of source embodiment. It pulls representations toward contact geometry and end-effector velocity rather than joint configurations.

Separate per-robot policies win when embodiments impose fundamentally incompatible action spaces. Consider a Boston Dynamics Spot (12-DOF legged locomotion, proprioceptive joint torques at 1 kHz, where DOF is the number of independently controllable joints) and a Franka Panda (7-DOF Cartesian impedance control, where the controller regulates end-effector position and compliant force response rather than individual joint angles, at a 500 Hz communication ceiling). They share a control rate but differ so deeply in contact physics and action dimensionality that a single canonical action vector distorts both.

Where Transfer Usually Breaks

A well-formed canonical interface lets gradients from one robot help another. It also fails at specific seams. Each seam corrupts an entry in the shared latent space instead of crashing, so they are worth cataloguing before they cost you a training run.

The table above is why the Open X-Embodiment RT-X paper devotes more space to dataset alignment than to model architecture, and why LeRobot v3 makes fps, action units, and camera topology mandatory schema fields rather than optional tags. Across the 22 RT-X robots, transfer was lost at exactly these four interface points, not at the optimizer.

A model trained on unit-inconsistent trajectories does not learn to grasp; it learns to decode the units.