Section 35.5: Adapting to new robots; prompting and conditioning

"A new robot is not a new prompt; it is a new contract with gravity, timing, and hardware limits."

An Adaptation Engineer

What Changes When The Robot Changes

As Figure 35.5A suggests, adapting to a new robot is less about giving the policy a new identity and more about snapping the right adapter plate between a shared prior and an embodiment-specific contract. Moving to a new robot can alter camera topology, frame conventions, control frequency, gripper state, actuator delay, and safety envelopes. Language conditioning may stay almost unchanged while the motor contract changes completely. That asymmetry is why adaptation pipelines usually begin with calibration and action remapping before they touch model weights.

The common mistake is to treat "prompting and conditioning" as if embodiment transfer were mostly a semantic problem. In reality, prompting is only one conditioning channel among many. Embodiment tokens, calibration metadata, action adapters, and low-rank fine-tuning all compete to absorb different parts of the shift. This ordering matters in practice. On an OpenVLA backbone moving from a Franka Panda to a WidowX arm, end-to-end fine-tuning through a mismatched action interface burned hundreds of demonstrations to recover task success. Inserting a linear action adapter on the WidowX joint-velocity range first cut that cost to tens of demonstrations (OpenVLA, Stanford / UC Berkeley, 2024). That is the interface-first rule in numbers, and it is the fastest argument against skipping calibration.

A Minimal Adaptation Equation

To act on the interface-first rule rather than merely assert it, we need a way to name each place the embodiment shift can be absorbed, which is what the following decomposition provides.

One useful decomposition is

$$a_t^{(r)} = A_r\big(\pi_{\theta + \Delta\theta}(E(o_t, m_r), q_t, e_r)\big),$$

where \(m_r\) is embodiment metadata, \(e_r\) is an embodiment token or descriptor, \(A_r\) is the robot-specific action adapter, and \(\Delta\theta\) is any fine-tuning update. Keep each term honest: metadata handles known configuration, tokens capture coarse robot identity, adapters handle command semantics, and weight updates cover only what genuinely needs learning. The diagram below traces the equation as a left-to-right data path, showing where each term acts as an observation becomes an action.

Consider a specific case: a shared policy trained on a 7-DoF (degrees of freedom) tabletop arm at 10 Hz transfers to a 6-DoF arm with a parallel-jaw gripper running at 25 Hz. The metadata term \(m_r\) encodes the new control frequency and gripper type as structured fields. The policy reads these directly rather than inferring them from context. The embodiment token \(e_r\) is a short learned descriptor, for example a 16-dimensional vector, that shifts the policy's latent distribution toward the new kinematic regime.

The token matters because a shared backbone trained across diverse robots learns an average motor behavior that is wrong for any single robot. Counterintuitively, a policy trained on 50 robots often performs worse out of the box on robot 51 than a policy trained on just 5 robots in the same kinematic family. The larger training set pulls the average further from any individual embodiment. Without a bias toward the target embodiment, the policy issues torques calibrated for an arm that never existed. On hardware, that mismatch produces overshoot, wrist instability, or gripper commands sized for the wrong jaw width. Those failures cause physical damage or task failure rather than merely lower accuracy. Mechanically, the system concatenates \(e_r\) with the observation embedding before the first transformer layer. During adaptation, gradient descent updates only \(e_r\) on a small demonstration set while the backbone stays frozen, so the token pulls activations toward the subspace the new robot occupies. The action adapter \(A_r\) rescales joint-velocity commands from the original normalized range to the new hardware limits and reorders the DoF indices to match the new URDF (Unified Robot Description Format, the XML file that specifies a robot's joints, links, and kinematic tree). Because the camera setup is unchanged, \(\Delta\theta\) is zero. In this case, adaptation needs no weight updates and typically costs a few minutes of calibration rather than hours of fine-tuning.

What happens when a team skips these steps and dials straight to fine-tuning? In practice, the policy receives corrupted action targets: velocities scaled for the wrong arm, DoF indices pointing at joints that do not exist on the new robot, and a chunk length that cuts motion trajectories in half. The fine-tuning loop then attempts to learn its way out of a broken interface, wasting demonstrations and often ending in hardware faults. The algorithm below exists precisely to prevent that sequence.

Code Fragment 1 turns that decomposition into a practical decision rule.

# Choose the cheapest adaptation path that matches the interface mismatch.
cases = [
    {"robot": "same_arm_new_task", "camera_changed": False, "action_changed": False, "new_language": True},
    {"robot": "new_gripper_same_arm", "camera_changed": False, "action_changed": True, "new_language": False},
    {"robot": "mobile_manipulator", "camera_changed": True, "action_changed": True, "new_language": True},
]

for case in cases:
    if not case["camera_changed"] and not case["action_changed"]:
        decision = "prompt or small data fine-tune"
    elif case["action_changed"] and not case["camera_changed"]:
        decision = "action adapter plus validation"
    else:
        decision = "adapter, calibration, and post-training data"
    print(f"{case['robot']}: {decision}")

same_arm_new_task: prompt or small data fine-tune
new_gripper_same_arm: action adapter plus validation
mobile_manipulator: adapter, calibration, and post-training data

The expected output is a routing table that chooses the lightest adaptation lever compatible with the actual source of mismatch. The important interpretation is that prompt-only adaptation is reserved for semantic drift on familiar hardware, while embodiment shifts route immediately toward adapters, calibration, or parameter updates.

Prompting, Embodiment Tokens, And LoRA Are Not Interchangeable

Now that each term in the adaptation equation has a distinct job, it helps to see the four levers side by side, so that the routing logic in the code above becomes a matter of matching each mismatch to the one lever built to repair it.