Section 23.4: Immersive/VR teleoperation (Open-TeleVision)

"The operator wanted depth perception. The robot delivered a network delay with excellent lighting."

A Stereoscopic Operator

Perception Loop

Tilt your head left and, two hundred milliseconds later, the robot across the room turns its cameras to match: in that gap between intention and image lives every reason immersive teleoperation succeeds or quietly corrupts your data. Teleoperation is a closed-loop control problem with a human inside the loop, as Figure 23.4A illustrates: the same act of immersive looking that helps the operator is only useful if visual feedback, body mapping, and latency are engineered together. The operator observes a rendered state \(\tilde{o}_t\), chooses a command \(u_t\), and receives delayed feedback \(\tilde{o}_{t+\Delta}\). If stereoscopic feedback reduces pose uncertainty but increases delay, the system must quantify the tradeoff. The loop diagram below traces this end-to-end path and decomposes the delay into its five stages.

Useful immersive systems therefore log not only robot actions, but also headset pose, rendered camera stream, frame drops, hand tracking confidence, and command timestamps. One term recurs through the rest of this section and is worth defining now: hand retargeting is the mapping from the operator's tracked hand or wrist pose to the robot's own joint commands, which is necessary because human and robot hands rarely share the same reach, degrees of freedom, or joint limits.

A demonstration recorded through a broken interface is not a human intention: it is a human intention filtered through noise, and the policy learns the noise too. The audit below treats frame drops and delay as episode labels. This is not a performance nicety; it determines whether the resulting demonstration should be considered expert data.

# Classify immersive teleoperation episodes using delay and visual stability.
# The labels help separate expert intent from interface-induced mistakes.
episodes = [
    {"id": "vr001", "delay_ms": 72, "dropped_frames": 1, "tracking": 0.98},
    {"id": "vr002", "delay_ms": 180, "dropped_frames": 9, "tracking": 0.81},
]

for episode in episodes:
    stable_video = episode["dropped_frames"] <= 3 and episode["tracking"] >= 0.95
    acceptable_delay = episode["delay_ms"] <= 100
    label = "clean" if stable_video and acceptable_delay else "interface-risk"
    print(episode["id"], label)

The expected output routes vr002 away from clean training because multiple interface signals are weak at once. This is the central engineering point: immersive teleoperation data should preserve the quality of the operator's perceptual channel. If a policy later fails on an episode labeled interface-risk, the team should inspect video delay, dropped frames, and hand tracking before blaming the manipulation model.

Those quality labels only matter because the immersive interface is doing something a fixed setup cannot, so it is worth examining exactly what extra information active visual feedback puts in the operator's hands.

Mechanism: Active Visual Feedback

Open-TeleVision-style systems change the information available to the operator. A fixed camera gives a passive view of the scene; active stereo feedback lets the operator move viewpoint, resolve occlusions, and align the robot body with task-relevant geometry. That extra perception can improve demonstrations for tasks such as opening drawers, threading tools, or reaching around clutter, where the critical state is not visible from one fixed camera.

Consider a specific case. An operator collecting drawer-opening demonstrations with a fixed overhead camera cannot see the handle recess from above, so every approach relies on an occluded view. A head-mounted stereo camera lets the operator tilt the viewpoint 30 degrees downward. This resolves the handle geometry and, in Open-TeleVision's reported evaluations on similar tabletop tasks, typically cuts failed grasp attempts from roughly 40% to under 10% (Cheng et al., 2024); exact numbers vary by task and operator, so treat the figures as illustrative rather than guaranteed. The information chain is direct. Stereoscopic parallax resolves depth at 20 to 50 cm range, the operator sees the handle lip clearly, the approach angle improves, and the recorded wrist trajectory carries that improvement into the training set. To put this concretely: under favorable conditions, a policy trained on 300 clean immersive episodes can approach the manipulation success rate of one trained on roughly 2,000 episodes collected under the same task with a fixed overhead camera, though this ratio is a rough illustration rather than a fixed conversion rate (the kind of data-scaling tradeoff studied directly in later chapters). Each immersive episode contains far fewer approach errors that the policy must later unlearn. The same benefit does not appear when the critical geometry is already visible from a fixed camera, so measuring per-task grasp success rate before and after adding immersive feedback beats assuming it always helps.

Where the operator looks is also data

Active feedback also introduces a second policy-like behavior: the human chooses not only hand motion, but also where to look, a capability called active gaze control. A dataset that keeps headset pose and camera motion can later train or evaluate active perception policies. A dataset that stores only wrist images discards the reason immersion helped.

What happens when the policy reaches a precision insertion step but has never learned to look for the target hole first? It fails repeatedly at exactly that moment, with no mechanism to recover, because the training data never captured the viewpoint shift that resolves it.

Active gaze control matters in embodied AI because a real robot cannot pause to re-examine a scene. It must gather visual evidence while acting. When an operator looks around intelligently but the system never records those gaze decisions, the learned policy receives only wrist trajectories. It gets no signal about which views resolved uncertainty before each move. On a physical robot, this gap causes failures at precisely the moment a task demands a deliberate viewpoint shift, such as before a precision insertion or after an occlusion. The policy never learned any information-gathering behavior, so it fails.

The mechanism is a pan-tilt camera mount or a gimbal on the robot's head linked to the headset's inertial measurement unit (IMU). As the operator turns their head, the IMU reports a rotation in the headset frame; the system converts that rotation to a motor command for the robot-mounted camera, completing a gaze-following loop typically at 30 to 90 Hz. The rendered stereo frames then reflect the new viewpoint with a delay set by the latency budget. Recording the headset pose stream alongside wrist images preserves both the motion and the information-gathering intent in the same episode.

Concrete Tool Anchors

The algorithm above assumes a stack that can timestamp, calibrate, and stream every one of those signals, so the next question is which concrete tools actually supply each layer.

A practical immersive stack layers four maintained components. Open-TeleVision supplies the reference design for active visual feedback and imitation-learning collection. A robot middleware layer such as ROS 2 carries robot state, commands, emergency-stop status, and camera topics. WebRTC-style streaming or a vendor headset SDK carries the visual channel. A calibration tool (Kalibr, OpenCV routines, or the platform's own stereo workflow) records the geometry that makes depth perception meaningful.

The important engineering decision is where each timestamp is created. Camera capture time, network receive time, headset render time, hand-tracking time, and robot command time should all be recorded or reconstructable. Without those anchors, a replay can look smooth while hiding the delay that shaped the operator's actions.