Section 23.2: Leader-follower teleoperation (ALOHA, GELLO)

"The best joystick for an arm is sometimes another arm that admits it is only pretending."

A Kinematic Twin

A teleoperation interface that forces the operator to solve geometry in their head is not a data collection tool; it is a cognitive tax on every demonstration in the dataset.

Kinematic Mapping

The cleanest leader-follower interface makes the leader configuration \(q_L\) correspond to a follower configuration \(q_F\). The simplest mapping is an affine calibration in joint space:

$$q_F = S(q_L - q_L^0) + q_F^0,$$

where \(q_L^0\) and \(q_F^0\) are neutral poses and \(S\) contains joint sign and scale terms. More complex systems map end-effector poses through forward and inverse kinematics, but the principle is the same: the operator should not mentally solve geometry that the device can embody. The diagram below traces this calibration from operator motion through the leader arm, the affine mapping, and into the follower arm.

ALOHA and GELLO apply this same affine mapping but differ in what the leader hardware is. ALOHA builds its leader from the same Low-Cost Robot Arm (LCRA) as the follower, so the calibration in Equation above is close to the identity map with a small sign and offset correction. GELLO takes the opposite starting point: it is an open-source, 3D-printed leader arm driven by Dynamixel servos (a common hobby-robotics actuator with built-in position and current sensing) that is explicitly designed to be a kinematic twin of a target commercial follower arm, such as a UR5 or a Franka Panda, that was not built with teleoperation in mind. Because a GELLO leader is manufactured to mirror the follower's link lengths and joint axes, the same \(S(q_L - q_L^0) + q_F^0\) mapping applies with \(S\) close to a signed identity, and a lab can build a matched leader for almost any commercial arm without redesigning the follower.

The affine joint-space mapping works well when leader and follower share the same kinematic structure. In ALOHA, both arms are Low-Cost Robot Arms (LCRA) with matched joint counts and similar link lengths, so the mapping is direct. The proficiency payoff is concrete. Operators using a kinematically matched leader typically reach proficiency in under 30 minutes. For the same task, operators using a joystick or a SpaceMouse (a six-axis input puck that translates hand push and twist into end-effector velocity commands) often need 5 to 10 hours of practice to match that success rate on dexterous bimanual tasks. That skill gap compounds into a data gap. A mismatched interface produces noisier demonstrations, so you need roughly 10 to 20 times as many episodes to train a policy of equivalent quality. The original ALOHA paper reached 90% task success with around 50 demonstrations per task. SpaceMouse-collected datasets on comparable contact-rich tasks typically required 500 to 1000 episodes to reach the same threshold. The mapping breaks down when the two arms differ in workspace extent or joint range. A leader wrist that rotates 270 degrees driving a follower limited to 180 degrees will hit a hard stop the operator cannot feel. That mismatch typically injects a clip artifact into demonstrations that approach the boundary, though the frequency depends on how often the task actually drives the joint near its limit. In those cases, a task-space mapping through forward and inverse kinematics is safer: building on the inverse-kinematics machinery from section 5.6, where a singularity is a configuration in which the arm loses a degree of freedom and the inverse-kinematics solution becomes ill-conditioned or non-unique, the system can detect the singularity and alert the operator before the follower executes it.

Latency And Stability

Kinematic matching removes the geometry the operator must solve in their head, but even a perfectly calibrated leader still corrupts labels if the signal arrives late, so the next axis of fidelity is timing rather than geometry.

Latency changes what the operator sees and what the robot executes, creating what is called the observe-execute time gap, which corrupts action labels. Let \(\Delta t = t_{execute} - t_{observe}\). At 50 Hz, a control step is 20 ms; at 5 Hz, it is 200 ms. For contact-rich manipulation, that difference can decide whether the operator corrects a slip or records an avoidable failure.

The following snippet computes a latency budget and flags episodes that should not be used as clean expert data.

# Audit leader-follower timing so bad labels are not treated as expert actions.
# Episodes over the latency threshold should be labeled or excluded from clean splits.
episodes = [
    {"id": "ep001", "control_hz": 50, "network_ms": 18, "camera_ms": 12},
    {"id": "ep002", "control_hz": 10, "network_ms": 55, "camera_ms": 40},
]

for episode in episodes:
    control_ms = 1000 / episode["control_hz"]
    total_ms = control_ms + episode["network_ms"] + episode["camera_ms"]
    label = "clean" if total_ms <= 80 else "latency-risk"
    print(episode["id"], round(total_ms, 1), "ms", label)

The output distinguishes an episode that can serve as clean supervision from one that needs a latency-risk label. That distinction matters because a delayed correction can look like a bad action in the dataset even when the operator made the right decision based on stale feedback. A serious collection pipeline keeps both rows, but routes them to different training or stress-evaluation uses.

Deadman Switches And Safety Interlocks

Timing budgets protect label quality. A second class of interlock protects the operator and hardware from the moments when control breaks down entirely. A deadman switch is a hardware or software interlock that halts follower motion the moment the operator releases a held button or grip. Without one, any hand slip, distraction, or cable snag lets the robot keep executing instead of freezing in its last commanded pose. That matters because a manipulator arm can exert enough force at speed to damage itself, the workspace, or a nearby person. In embodied AI data collection, the stakes extend beyond hardware safety. An unintended motion produces an action label that looks like expert behavior but is actually noise, and the pipeline then bakes that noise silently into the training set.

The mechanical version wires the switch in series with the control-enable signal: releasing it opens the circuit, and the follower gets a zero-velocity command on the next cycle. The software version runs a watchdog timer (a countdown that resets on every expected signal and triggers a fail-safe action if it is not reset in time) that expects a fresh "active" heartbeat every control period, and clamps commanded velocities to zero when the heartbeat is late. Both stop the robot within one control cycle instead of letting an unsafe partial motion finish.

The table below summarizes the four design axes that most often separate a leader-follower rig that produces clean expert data from one that quietly corrupts it, pairing each axis with the observable signal that tells you which side you are on.