"Every dataset has a junk drawer. The question is whether you labeled it."
A Patient Data Curator
This section assumes familiarity with the teleoperation hardware and episode structure introduced in sections 23.1 through 23.3. The quality and labeling decisions made here feed directly into the dataset format covered in section 23.6 and the scaling analysis in section 24.1. The failure-cause taxonomy recurs in section 27.7, where perception errors are isolated as a specific failure mode, and in section 26.4, where instruction labels become the conditioning signal for language-conditioned policies.
A robot learns to grasp a mug. Fifty episodes land in a folder. Six have dropped camera frames. Eight end with the operator grabbing the arm. Three succeed only because the reset happened to place the mug closer than intended. If all fifty feed the policy equally, the model quietly learns from corrupted geometry, from human rescue moves it will never receive at test time, and from a distribution it will never see again. Large-scale robot datasets are exposing this problem at scale right now: raw teleoperation hours do not automatically become useful training data. In one representative audit (circa 2023-2024), 100 collected episodes yielded only 61 after quality gating; the policy trained on the curated 61 outperformed the policy trained on all 100 by roughly 20 percentage points on held-out tasks, because the discarded episodes were teaching the wrong distribution. Knowing how to score, route, and label each episode, before anything reaches the trainer, is the skill that separates researchers who scale gracefully from those who wonder why more data made things worse. You will build a quality-gate pipeline and a typed failure taxonomy that turns a messy episode folder into deliberate train, evaluation, and repair splits.
Quality As A Labeling Problem
Two episodes both end with the robot dropping the mug, and to a success-only filter they look identical. Yet a camera bumped out of calibration doomed one, and a gripper that closed a fraction too late doomed the other. Collapse them into a single "fail" label and you throw away the only information that tells you which repair to make. As Figure 23.5A captures, a quality-gate stage sits between raw demonstrations and the train, validation, stress, and repair assets that follow. An episode has at least three truth layers: what the operator intended, what the robot executed, and what the environment actually did. Labels should distinguish these layers. "Failure" is too coarse; "object slipped after contact", "operator intervened", "camera dropped frames", and "reset distribution mismatch" teach different lessons.
A useful data-quality score can be written as a weighted checklist:
$$Q(e) = w_s S(e) + w_c C(e) + w_t T(e) + w_l L(e) - w_i I(e),$$
where \(S\) is task success, \(C\) calibration validity, \(T\) timing validity, \(L\) label completeness, and \(I\) intervention burden. The weights should be stated in the dataset card rather than rediscovered from code.
Why Calibration Drift Poisons a Whole Episode
Calibration validity matters because a robot policy learns a mapping from camera pixels to joint commands. If the hand-eye calibration (the measured geometric relationship between the camera frame and the robot's wrist or base frame) drifted between the calibration session and the collection session, the drift offsets every pixel-to-world transform in that episode. The policy trains on phantom object positions; at deployment, the real robot reaches for air. A single episode collected after a camera was bumped can silently bias thousands of gradient steps unless a gate flags it and routes it to repair. Take a typical 30 Hz, 200-step episode. That one corrupted recording contributes 6,000 training samples. Every pixel-to-world lookup carries the same phantom offset, and those samples outvote many clean episodes before the optimizer ever sees correct geometry.
Checkpoint
So far: labels need to distinguish truth layers instead of collapsing to one "fail" class, the score \(Q(e)\) combines success, calibration, timing, label completeness, and intervention burden into one weighted number, and calibration drift matters because it silently poisons every pixel-to-world transform in an episode, not just one frame. Next, the section turns this mechanism into an automated check.
The check works by comparing the calibration timestamp stored in each episode's metadata against the most recent validated calibration file for that robot serial number. If the gap exceeds a configured threshold, the episode sets \(C = 0\). This comparison requires nothing more than a datetime lookup, but it must be automated: manual review cannot catch a three-hour-old calibration in a folder of fifty episodes without a dedicated query.
In practice, applying this gate to a raw episode folder follows a short, repeatable sequence: (1) for each episode, read the stored calibration timestamp and diff it against the robot's most recent validated calibration file; (2) if the gap exceeds the configured threshold (commonly 1 to 2 hours for tabletop rigs, tighter for high-precision tasks), set \(C(e) = 0\) for that episode and record the reason in the manifest; (3) fold \(C(e)\) into \(Q(e)\) alongside \(S\), \(T\), \(L\), and \(I\) using the weighted formula above; (4) let the resulting score, not a human's spot check, decide the split. Section 23.6 shows how LeRobotDataset stores this calibration timestamp so the check can run as a single Parquet query rather than a manual folder scan.
Figure 23.5B shows how these checks combine: raw episodes pass through the quality gates, receive a per-episode score \(Q(e)\), and then route to the train, stress, or repair split according to threshold bands. The diagram traces the same pipeline this section describes in prose: a raw episode enters, passes the four gates, receives its score, and lands in exactly one of the three destination queues.
A failed rollout can be more useful than a success if it localizes the missing skill. Label the mechanism of failure, not only the outcome.
Use dataset validators, Pydantic schemas (Python classes that declare required fields and types, then raise an error if a record violates them), and LeRobot metadata checks to automate the boring parts of quality control. Human review should be reserved for semantic labels, ambiguous failures, and split decisions that require task knowledge.
The following example scores episodes and routes them into splits. Notice that the rule keeps stress data rather than deleting every imperfect row.
# Route episodes by quality gates instead of using one vague keep flag.
# Stress and repair examples stay useful because their failure type is explicit.
episodes = [
{"id": "a", "success": 1, "calibrated": 1, "synced": 1, "labels": 1, "interventions": 0},
{"id": "b", "success": 0, "calibrated": 1, "synced": 1, "labels": 1, "interventions": 2},
{"id": "c", "success": 1, "calibrated": 0, "synced": 0, "labels": 0, "interventions": 0},
]
for e in episodes:
score = e["success"] + e["calibrated"] + e["synced"] + e["labels"] - 0.5 * e["interventions"]
route = "train" if score >= 3.5 else "stress" if e["labels"] else "repair"
print(e["id"], score, route)score = success + calibrated + synced + labels - 0.5 * interventions and printing the resulting train/stress/repair route for episodes a, b, and c.The expected output sends episode a to training, episode b to stress evaluation, and episode c to repair. That routing is the mechanism behind useful curation: the pipeline does not ask whether an episode is good or bad in the abstract. It asks what scientific role the episode can play after calibration, synchronization, labels, and interventions are known.
Step-Through: Scoring And Routing Three Episodes
Trace the quality score \(Q(e) = S + C + T + L - 0.5 I\) (using unit weights and intervention weight 0.5) for three concrete episodes, then apply the bands train (\(Q \geq 3.5\)), stress (\(2 \leq Q < 3.5\) with labels present), and repair (otherwise).
Episode a (clean grasp): \(S=1\), \(C=1\), \(T=1\), \(L=1\), \(I=0\). So \(Q = 1 + 1 + 1 + 1 - 0.5 \times 0 = 4.0\). Since \(4.0 \geq 3.5\), route to train.
Episode b (task failed, operator grabbed the arm twice, but fully labeled): \(S=0\), \(C=1\), \(T=1\), \(L=1\), \(I=2\). So \(Q = 0 + 1 + 1 + 1 - 0.5 \times 2 = 2.0\). Since \(2.0\) falls in \([2, 3.5)\) and labels are present, route to stress: it documents where the policy needs human rescue.
Episode c (succeeded by luck, but stale calibration, dropped frames, no labels): \(S=1\), \(C=0\), \(T=0\), \(L=0\), \(I=0\). So \(Q = 1 + 0 + 0 + 0 - 0 = 1.0\). Since \(1.0 < 2\) and \(L=0\), route to repair: the apparent success is built on corrupted geometry and cannot be trusted until reviewed. Note how the lucky success scores far below the labeled failure, exactly the inversion that naive success-only filtering misses.
Annotation Schema
Routing an episode to train, stress, or repair is only as good as the labels feeding the score, so the next question is what minimum set of fields every episode must carry for that routing to mean anything.
| Field | Examples | Use |
|---|---|---|
| Task outcome | success, partial, fail, abort | Evaluation and filtering. |
| Failure mechanism | slip, collision, missed grasp, timeout, perception error | Error analysis and recovery training. |
| Intervention | none, human correction, emergency stop | Safety and autonomy measurement. |
| Data health | synced, dropped frames, calibration stale | Quality routing. |
| Instruction | language command, goal image, task id | Language-conditioned policy training. |
Consider a specific case: the DROID dataset (Khazatsky et al., 2024), which collected over 76,000 robot trajectories across 564 scenes, annotates each episode with a language instruction, a success flag, and a scene identifier. Open X-Embodiment is a large, cross-institution compilation that pools dozens of separately collected robot datasets into one shared training corpus, and RT-X refers to the family of policies trained on that pooled corpus. The Open X-Embodiment compilation goes further, requiring each contributing dataset to declare sensor modalities, action space, and control frequency in a shared metadata schema. Neither dataset treats failure as a single negative class: DROID separates human-aborted episodes from autonomous failures, and Open X-Embodiment preserves the original per-dataset failure taxonomies rather than flattening them. The lesson is that schema choices made at collection time directly constrain what analyses are possible years later, after the hardware is gone.
- Validate timestamps and stream lengths.
- Check calibration version against the collection session.
- Run automated label sanity checks.
- Manually inspect a stratified sample by task, operator, and outcome.
- Freeze split assignment and save the manifest hash (a fingerprint of the exact train/stress/repair file list, so a later re-run can prove no episode silently moved between splits).
Before reading on, consider: if two episodes both end in the robot dropping the object, but one was caused by a miscalibrated camera and the other by insufficient gripper force, how many different repairs do you need?
Mechanism: Separating Outcome From Cause
A dataset without cause labels is a graveyard of outcomes: you know what died, but not why, and the same mistakes keep recurring. Outcome labels answer whether the task succeeded. Cause labels answer why it succeeded or failed. A robot can fail because the perception system localized the object incorrectly, because the gripper command saturated, because the operator intervened late, or because the reset placed the object outside the intended distribution. These cases should not be merged into one negative class because they imply different repairs.
In training, cause labels drive recovery data, filtering, and curriculum design. In evaluation, they enable per-failure reporting: a policy might cut missed grasps while leaving occlusion failures untouched. That diagnosis beats a lone aggregate score.
This is not merely a recommended practice; the largest pooled robot datasets were built on exactly this refusal to collapse causes, as the following case shows.
Real-World Application: Open X-Embodiment
The Open X-Embodiment compilation (Padalkar et al., 2023) merged more than 60 robot datasets into one training corpus, and it refused to flatten failure into a single negative class: each contributing dataset declares its own sensor modalities, action space, and control frequency in a shared metadata schema, with per-dataset failure taxonomies preserved intact. That schema discipline is exactly why the RT-X models trained on the pooled data could be evaluated per-embodiment and per-failure-type rather than on one opaque aggregate success number.
Reporting "failure" without a cause is like returning a car to the mechanic and explaining that it "did not go." Technically accurate, almost entirely unhelpful, and a near-perfect description of how aggregate evaluation scores have been used in robot learning for a decade.
A common assumption is that collecting data across many scenes, operators, and object configurations guarantees a high-quality dataset. In embodied AI, this assumption typically does not hold, because surface diversity does not by itself remove systematic corruptions. A camera bumped between calibration and collection produces phantom pixel-to-world transforms in every episode, regardless of how many scenes were covered. Operator interventions captured alongside clean trajectories teach the policy rescues it will never receive at test time. As a result, more data of mixed integrity can actively hurt policy performance rather than help it. In one representative audit (circa 2023-2024), training on 61 curated episodes outperformed training on 100 uncurated ones by roughly 20 percentage points; the size of that gap will vary with corruption rate and task, but the direction, curated-smaller beating raw-larger, recurs whenever mislabeled or miscalibrated episodes are common enough to outvote clean ones. Diversity controls the distribution the policy sees. Quality gating controls whether each episode teaches the right mapping. Both are necessary, and neither substitutes for the other.
Annotation lag is like a football referee who blows the whistle the moment the ball hits the ground, when the foul that caused the fumble happened two steps earlier. The visible event (ball on the ground, object slipping) is downstream of the actual cause (the illegal contact, the insufficient grip force). If you label the frame where the failure becomes obvious, you are teaching a recovery policy to react to consequences it can no longer prevent, not to the early signals that actually predicted trouble.
Cause labels assigned after the fact suffer from annotation lag: the human reviewer watches the video and marks the failure at the moment it becomes obvious, which is often several frames after the actual causal event (a grasp that was already insufficient before the slip became visible). If those frame-level labels are used to train a recovery policy, the policy learns to respond to visible consequences rather than early signals. Validate temporal label alignment by checking whether the annotated frame is upstream or downstream of the first observable precursor in the sensor stream.
When assigning frame-level cause labels in LeRobotDataset, subtract a fixed pre-event offset from the annotated timestamp field rather than marking the frame where the failure becomes visible. A starting offset of 0.3 to 0.5 seconds (roughly 9 to 15 frames at 30 Hz) reliably places the label upstream of the visual consequence for most contact failures such as slips and missed grasps. Store the chosen offset as a dataset-card field named label_precursor_offset_s so that downstream users know the convention and can adjust it when resampling to a different control frequency.
Do not place near-duplicate episodes from the same collection burst into both train and validation. The model can appear to generalize while merely repeating a neighboring trajectory.
For a bin-picking dataset, the validation split should hold out object instances or clutter layouts, not merely every tenth video. Otherwise validation measures replay familiarity rather than deployment readiness.
Automated data quality scoring at scale. As robot fleets grow, hand-curated episode review becomes a bottleneck. The DROID dataset team (Khazatsky et al., 2024) documented how even a single corrupted calibration session can silently bias tens of thousands of gradient steps, motivating automated quality metrics that flag calibration drift, sensor dropout, and reset-distribution anomalies without human inspection of every clip. Active work in 2024-2025 at Stanford, CMU, and Berkeley is extending these checks to multi-robot deployments where operator skill variance is a first-class quality signal.
Targeted active collection driven by policy failure maps. Rather than collecting uniformly across scenes, recent work uses policy uncertainty or contact-force residuals from prior rollouts to identify underrepresented regions of the action distribution. The Physical Intelligence (pi) team's experiments with diverse manipulation policies (Black et al., 2024) showed that coverage of rare contact configurations, not raw episode count, predicted downstream generalization, motivating algorithms that translate policy failure logs into targeted operator task queues.
Automatic cause labeling from multimodal sensor streams. Human annotation of failure mechanisms is slow and suffers from annotation lag. Research in 2024-2026 is training small auxiliary models on contact forces, gripper torque, and vision to produce typed cause labels (slip, missed grasp, occlusion, reset mismatch) automatically during collection, removing the offline labeling step entirely. The Humanoid Locomotion as Next Token Prediction work (Radosavovic et al., 2024) demonstrated that richer automatic annotation of trajectory segments directly improved downstream policy learning without added operator time.
Open PhD problem. No principled method yet exists for deciding when a dataset's failure-cause distribution is complete enough to stop targeted collection. Current practice uses fixed episode budgets or operator fatigue as the stopping criterion. A student could formalize this as a coverage stopping rule: given a typed failure taxonomy and per-type episode counts, estimate the expected policy improvement from one more batch of targeted collection versus zero, and stop when the marginal value falls below a hardware-cost threshold. This requires combining Bayesian data-value estimation with robot-specific failure-mode models, and no published benchmark exists to evaluate such a stopping rule fairly.
Can each failure in your dataset be routed to perception, calibration, action representation, timing, contact dynamics, or task specification? If not, the labels are too coarse for serious improvement.
Data quality is not the absence of failures. It is the presence of typed, synchronized, split-aware evidence that tells the learner and the researcher what each episode means.
Write five failure labels for a pouring task and specify which labels belong in train, stress validation, or repair queues.
Lab: Watch Curation Beat Raw Volume
Goal. Empirically reproduce the central claim of this section: a small curated split can outperform a larger uncurated one, because quality controls which mapping the policy learns.
Tools needed. Python with scikit-learn and NumPy (no robot or GPU required). The "policy" is a small regressor; the "episodes" are synthetic samples, so the whole experiment runs on a laptop.
Setup. Generate 100 "episodes", each a batch of (image-feature, target-action) pairs drawn from a clean linear mapping plus noise. Mark 39 of them as corrupted: for those, add a fixed phantom offset to the input features (simulating stale hand-eye calibration) so the input-to-action mapping is consistently wrong. Compute a \(Q(e)\) score per episode from a calibration flag and a frame-drop flag, route \(Q \geq 3.5\) to a curated set, and keep a held-out clean test set that no episode-corruption touches.
What to vary. (1) Train one regressor on all 100 episodes and another on the roughly 61 curated episodes. (2) Sweep the corrupted fraction from 0 to 0.5. (3) Sweep the phantom-offset magnitude from small to large.
What to observe. Plot held-out test error for both models against corrupted fraction. You should see the all-data model degrade steeply as corruption rises while the curated model stays flat, and the two curves crossing near the corruption level where the phantom-offset episodes start outvoting clean geometry. This reproduces, in miniature, the roughly 20-point gap reported for 61 curated versus 100 uncurated robot episodes.
Project Ideas
Beginner (weekend): Episode quality scorer with LeRobot. Build a command-line tool that reads a LeRobotDataset folder, computes Q(e) scores using timestep counts, calibration metadata, and a simple intervention-flag field, and prints a routed manifest (train / stress / repair). The key challenge is parsing LeRobotDataset's Parquet episode tables and matching calibration timestamps without assuming a fixed schema version.
Intermediate (1 to 2 weeks): Failure taxonomy labeler in PyBullet or MuJoCo. Instrument a simulated pick-and-place task so that each rollout automatically records a structured failure label (slip, missed grasp, collision, timeout, reset mismatch) by inspecting contact forces and joint-limit flags, then exports a labeled Hierarchical Data Format 5 (HDF5) dataset compatible with LeRobot. The key challenge is translating raw physics-engine contact reports into semantically meaningful, policy-relevant failure categories that align with the typed taxonomy in this section.
What's Next
Section 23.6 shows how a standardized dataset format, especially LeRobotDataset, turns these quality decisions into reusable files and metadata.
Defines the handheld gripper approach, latency matching, and relative-trajectory action interface used in portable demonstration collection.
Cheng, X. et al. (2024). Open-TeleVision: Teleoperation with Immersive Active Visual Feedback.
A current reference for immersive visual feedback, active perception, and VR-style operator embodiment in data collection.
Zhao, T. Z. et al. (2023). Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware.
Introduces ALOHA and ACT, making the connection between low-cost bimanual teleoperation, action chunking, and real-world manipulation data explicit.
A kinematically matched leader device study that directly compares teleoperation ergonomics and reliability against other low-cost interfaces.
Hugging Face LeRobot Documentation.
Documents dataset conversion, policy training, and robot-control utilities that turn teleoperation logs into reusable learning artifacts.