Section 46.5: Teleoperation for humanoids

"Teleoperation is not a fallback. It is one of the fastest ways to reveal what the autonomy stack does not yet know how to do."

A Shared-Autonomy Design Review

A third of a second: that is roughly how stale the world already looks by the time an operator's command reaches a humanoid across a network, and it is exactly why you cannot puppeteer a robot's balance no matter how skilled you are. Teleoperation earns its place not by dodging that delay but by exposing it, because, as Figure 46.5A shows, the same link that builds data and validates interfaces also catches safety gaps long before fully autonomous deployment. A useful teleoperation latency budget is \(T_{\mathrm{total}} = T_{\mathrm{sense}} + T_{\mathrm{encode}} + T_{\mathrm{network}} + T_{\mathrm{render}} + T_{\mathrm{human}} + T_{\mathrm{robot}}\). For high-bandwidth whole-body tasks, that sum shapes what can be directly operated and what must be handed to autonomous stabilizers or motion primitives.

Shared autonomy can be written as \(u = \alpha u_{\mathrm{human}} + (1 - \alpha) u_{\mathrm{auto}}\), but the real system is richer. The human may command task intent while the robot closes local balance, collision, or grasp-stability loops. The best teleoperation interfaces expose this division clearly.

Why the partition is not optional

The partition matters physically because a humanoid's mechanical stability is governed by time constants far shorter than any human reaction loop. Letting \(\alpha = 1\) (pure human control) over balance degrees of freedom at typical network delays causes falls; the physical robot cannot wait for a correction that arrives 250 ms late. Getting this wrong destroys hardware, creates safety hazards, and produces training data contaminated by recovery motions rather than skilled intent.

In practice, \(\alpha\) is set per degree of freedom and per task phase rather than as a global scalar. Balance and contact joints hold \(\alpha\) near zero at all times. Wrist and finger joints may run at \(\alpha = 1\) during precision insertion. The weight can also be gated dynamically: if predicted contact force deviates beyond a threshold, the onboard controller temporarily overrides the human command for that joint only, then hands control back once stability is restored. That predicted contact force typically comes from the robot's own force or torque sensors at the wrist and fingertips, filtered forward a few control cycles by the same low-level controller that already runs the local balance and grasp loops, so the gating logic reuses infrastructure the robot needs anyway rather than adding a separate prediction model.

Theory

Figure 46.5.1 lays out the full loop: observe operator and robot state, model the shared-autonomy partition, act by blending human intent with onboard stabilizers, and verify success alongside workload and interventions. Teleoperation is a productive first-class research layer because it solves three problems at once: it provides coverage for hard tasks, a direct debugging path for failed autonomy, and a data stream for imitation or behavior modeling.

Humanoid teleoperation is especially demanding because whole-body motion, balance, and manipulation are tightly coupled. A human operator may specify intent, but the local stabilizer still has to protect contact feasibility and safety zones. The gap in practice is stark. A humanoid balancing loop needs a response within 20 ms to stay stable. A typical teleop round-trip runs 250-300 ms, more than ten times too slow for direct balance control. Balancing a humanoid over a 300 ms teleop link is like catching a falling glass while wearing oven mitts and looking through a window that shows the room as it was a third of a second ago. That mismatch is why essentially every deployed humanoid teleop stack we are aware of offloads balance and contact stabilization to onboard autonomy by design, not as a convenience.

Evaluation should therefore track not only task success but also operator workload, intervention frequency, takeover time (the delay between an autonomy failure and the operator regaining effective control of the affected joints), packet delay, and the fraction of control handled autonomously.

Real deployments illustrate how different teams draw the autonomy boundary. The 1X NEO system uses expert-mode supervision, a form of supervisory teleoperation in which an operator issues high-level task commands rather than direct joint commands. Operators command high-level actions while the robot closes local balance and contact loops at full servo rate. This keeps end-to-end human latency out of the stability-critical path. NVIDIA GR00T Whole-Body Control reports a similar split. The teleop interface sends Cartesian hand targets (the desired hand position and orientation in 3D space, rather than a target angle for each individual joint) while an onboard whole-body controller solves for joint trajectories and footstep feasibility in real time. Stanford HumanPlus (Fu et al. 2024) retargets full-body motion-capture pose to the robot at roughly 50 Hz. The robot's built-in balance stabilizer absorbs pose tracking error, so the operator never commands it explicitly.

Worked Example

To see how the latency budget and autonomy fraction turn into an actual design call, it helps to run the numbers on a single session. A small teleop run summary can reveal whether failure came from latency, viewpoint, or missing autonomy support rather than from human skill.

<span>latency_ms = {"sense": 18, "encode": 8, "network": 42, "render": 25, "human": 180, "robot": 14}</span>
total = sum(latency_ms.values())
autonomy_fraction = 0.62
print({"total_latency_ms": total, "autonomy_fraction": autonomy_fraction})
print({"direct_teleop_ok_for_fast_balance": total < 120})

Expected output interpretation. At 287 ms end-to-end latency, direct whole-body balance control is unrealistic. The operator can still command intent, but stabilization must be handled by local autonomy or motion primitives.

Practical Recipe

1. Measure the real latency budget before choosing control granularity.
2. Move fast stabilization to the robot side when latency exceeds the task envelope.
3. Log operator intent and autonomous corrections separately.
4. Turn teleoperation traces into dataset artifacts rather than disposable operator sessions.
5. Review the top recurring interventions every week and convert them into automation candidates.

Once the partition, tooling, and deployment patterns above are in place, the payoff is a shift in how you think about the whole activity. The key reframe is to treat teleoperation as instrumentation rather than as failure. The best teams mine teleop traces for control bottlenecks, perception blind spots, and policy interface mistakes.

Teleoperation quality also sets a ceiling on data scaling: it governs not just task success in the moment but the fidelity of every demonstration that later trains an autonomous policy.

Teleop failures should be labeled by latency, viewpoint, operator overload, shared-autonomy mismatch, or low-level robot instability. Only one of those is fixed by training the operator harder.