Section 23.3: Handheld and in-the-wild collection (UMI)

"I collected the demonstration in a kitchen. The robot insists it has never been to one."

A Portable Gripper

A demonstration collected anywhere in the world is only as portable as the calibration discipline that traveled with it.

From Human Motion To Robot Action

The handheld unit itself is a stripped-down gripper mounted with a wide-angle egocentric camera and, in most builds, a fisheye mirror or side mirrors that expand the field of view; a human operator carries it by hand and performs the task exactly as a robot's end-effector would, with no arm, base, or actuator attached. That physical setup is what makes "in-the-wild" collection possible: the sections below explain what the unit must record (relative pose, gripper width, video), what discipline keeps that recording usable (calibration, latency matching), and what test proves it transferred correctly (robot replay), so that by the end of this section a captured session can be judged ready or not ready for policy training.

What if the most valuable thing you record in a kitchen is not where your hand was, but how far it moved between two instants? That single shift, from absolute pose to relative trajectory, is the abstraction that lets a demonstration filmed on a coffee-shop counter run on a robot that has never left the lab. Instead of asking the policy to imitate absolute human pose, the dataset stores local end-effector displacements over a short horizon. The payoff is concrete: shipping a robot to a new site costs weeks and thousands of dollars; a handheld gripper fits in a backpack and reaches the same site in an hour, so a team of five people can collect demonstrations across twenty kitchens in a single day instead of a single lab in a single month. A dataset that would have required 18 months of robot-site scheduling to reach 500 diverse episodes can be assembled in two weeks of handheld collection across 30 real environments, because the bottleneck shifts from robot availability to human walking speed.

$$\Delta x_{t:t+H} = (T_t^{-1}T_{t+1}, T_t^{-1}T_{t+2}, \ldots, T_t^{-1}T_{t+H}),$$

where \(T_t\) is the gripper pose at time \(t\). Relative actions cut the dependence on a specific room coordinate frame. A robot with a different base pose can therefore run the learned behavior directly. Figure 23.3A captures this separation: humans demonstrate in one place, robots deploy in another, and the relative-action representation bridges the gap. The workflow diagram below traces the same idea as a three-stage pipeline, from handheld collection through processing to robot deployment.

Code Fragment 1 shows a compact version of the relative-action transform used in many trajectory datasets. The example uses one-dimensional poses so the arithmetic is inspectable, but the same idea extends to full SE(3) transforms from Chapter 4.

# Convert absolute gripper positions into local relative actions.
# This teaches the representation before a full SE(3) trajectory library hides it.
positions_m = [0.40, 0.43, 0.47, 0.46]
relative_actions = []

for current, nxt in zip(positions_m, positions_m[1:]):
    relative_actions.append(round(nxt - current, 3))

print(relative_actions)
print("largest step:", max(abs(x) for x in relative_actions), "m")

Data Collection Recipe

Calibration discipline gets a demonstration's geometry right, but geometry is only half the bridge; the other half is timing, and of every field in the collection recipe, latency is the one whose mishandling fails most silently. Latency matching deserves special attention because its effect is counterintuitive. A policy is a function that maps an observation to an action. If the observation was always captured 65 ms before the action was executed during collection, then the policy learns to anticipate 65 ms into the future. At deployment, if the pipeline delivers observations 130 ms before execution, the policy now anticipates twice as far and systematically overshoots. The fix is not to minimize latency but to make it consistent, a principle called latency-invariant policy training: measure the end-to-end delay in your robot stack, then replay that exact delay during training by artificially offsetting the observation timestamps in the dataset. This is why step 4 in the recipe below is not a performance optimization but a correctness requirement.

1. Record synchronized egocentric video, gripper pose, gripper width, and task instruction.
2. Calibrate gripper geometry and camera extrinsics before each session.
3. Store both absolute sensor poses and relative policy actions.
4. Match inference-time latency during training so the policy sees the same delay it will experience on hardware.
5. Validate with a robot replay protocol before counting the episode as deployable data.

Why Replay Before You Trust the Manifest

The robot replay protocol matters because handheld collection opens a gap that only physical execution can close. A human operator compensates for hand tremor, calibration drift, and frame-rate irregularities in ways the recorded trajectory never captures. When those compensations corrupt the relative actions, the episode passes the manifest but fails on the arm. The cost is asymmetric: one silent bad episode trains the policy on physically impossible transitions, and engineers then blame model capacity instead of data quality.

In practice, the protocol commands the recorded relative action sequence straight to the robot arm in open loop (the controller executes the pre-recorded actions blindly, without adjusting based on new sensor feedback), with no policy inference. The arm executes each \(\Delta x_t\) step. The controller then compares the end-effector position against the expected pose from the original trajectory. Tracking error above a task-specific threshold (typically 2 to 5 mm for pick-and-place tasks) flags the episode for recollection. Open-loop replay isolates data quality from policy quality: if the robot cannot reproduce the recorded motion at all, discard the episode rather than pass it to training.

pip install pandas

# Build a handheld-demonstration manifest with one deliberate audit field.
# The example includes the stress condition explicitly so the audit can run end to end.
episodes = [
    {"id": "umi_001", "camera_calibrated": True, "latency_ms": 65, "split": "train"},
    {"id": "umi_002", "camera_calibrated": "latency_review", "latency_ms": 140, "split": "stress"},
]
print(episodes)

# Apply a readiness rule that separates clean data from stress or repair data.
for episode in episodes:
    ready = episode["camera_calibrated"] is True and episode["latency_ms"] <= 100
    episode["split"] = "train" if ready else "review"
summary = {split: sum(ep["split"] == split for ep in episodes) for split in {"train", "review"}}
print(summary)

# Complete manifest audit for handheld demonstrations.
# It keeps clean training data separate from risky but useful stress examples.
episodes = [
    {"id": "umi_001", "camera_calibrated": True, "latency_ms": 65, "split": "train"},
    {"id": "umi_002", "camera_calibrated": False, "latency_ms": 140, "split": "stress"},
]
for episode in episodes:
    ready = episode["camera_calibrated"] is True and episode["latency_ms"] <= 100
    route = "clean-train" if ready else "stress-or-recollect"
    print(episode["id"], route)