"A million trajectories sound large until you ask how many robot bodies they remember."
A Dataset Cartographer
This section assumes familiarity with how robot demonstrations are collected and structured, covered in section 23.1. The dataset properties surveyed here, particularly embodiment diversity and annotation richness, become the direct inputs to data-scaling analysis in section 24.3 and to cross-embodiment pretraining in section 35.2. Readers who already work with a specific dataset daily and want to focus on what makes a dataset scientifically useful may skip to section 24.2 for the formal dataset card schema.
In 2023, a single consortium pooled trajectories from 22 different robot bodies and over 500 tasks into one release. The bet behind that effort is now the central wager of embodied AI: if a policy sees enough physical diversity, it will generalize to bodies and scenes it has never touched. The datasets surveyed here, Open X-Embodiment, DROID, BridgeData V2, RH20T, RoboMIND, and AgiBot World, are competing answers to what "enough diversity" actually means. By the end of this section, you will be able to read a dataset card critically and choose the right corpus for a specific generalization claim.
Dataset Families
Two datasets can both advertise a million trajectories and still teach a policy opposite lessons: one might be a million repetitions of the same arm in the same kitchen, the other a thousand bodies each seen a thousand times, and only the second buys you the generalization you were paying for. That gap between repetition and coverage is why the datasets surveyed here differ not only in raw size but in robot bodies, scenes, tasks, annotations, and split philosophy, and those five axes are what a reader learns to read here. As Figure 24.1A illustrates, Open X-Embodiment pools many robots and tasks to study RT-X style cross-robot transfer (the mechanics of how heterogeneous robot recordings are actually merged into one trainable format are explained two paragraphs below, once the diversity axes are on the table). DROID emphasizes in-the-wild collection across diverse scenes and data collectors. BridgeData V2 emphasizes large-scale manipulation trajectories with language and goal-conditioning compatibility. RoboNet is historically important because it pushed multi-robot video prediction and self-supervised interaction data before the current foundation-model wave. The scatter plot below places these datasets on two of the diversity axes at once, embodiment diversity against scene diversity, and no single corpus dominates every direction.
Embodiment diversity matters because a policy learned on a single robot body tends to overfit its specific kinematics. The network learns that certain joint configurations predict good grasps on a 7-DOF (degrees of freedom) Franka. Those features do not transfer to a 6-DOF UR5 or a parallel-jaw gripper with a different reach envelope. When a robot fails in a new body, the cause is often not missing task knowledge but missing motor-level invariance. Pooling trajectories across bodies forces the network to discard body-specific shortcuts. It retains only those features that hold across multiple physical configurations, which is what makes downstream transfer feasible at all.
Pooling works through a shared episode schema. Each contributor converts their robot's collected demonstrations into RLDS (Robot Learning Dataset Standard) format. RLDS stores per-step dictionaries with camera images, proprioceptive state, and action vectors, plus episode-level metadata that records robot type and task source. Action vectors differ in dimension and units across bodies. To handle this, cross-embodiment policies take one of two routes. They either send each robot's actions through a learned per-robot linear head before a shared trunk, or tokenize continuous actions into discrete bins so the trunk sees a uniform vocabulary. The dataset exposes the incommensurable action spaces; the architecture must resolve them. The payoff is concrete. A single-robot policy typically needs around 50,000 demonstrations to reach reliable performance on a target task. A cross-embodiment policy pretrained on pooled data from 22 bodies reaches the same threshold with roughly 300 target-robot demonstrations, because the shared trunk arrives already knowing how to grasp, avoid obstacles, and interpret scene geometry.
Checkpoint
So far: a policy overfits to a single robot's kinematics unless training forces it to discard body-specific shortcuts; datasets enable this by converting every contributor's demonstrations into a shared RLDS episode schema; and because action vectors still differ in dimension and units across bodies, the architecture, not the dataset, resolves this mismatch through either per-robot action heads or a shared discrete action vocabulary.
If the architecture must resolve incommensurable action spaces, what actually happens when it encounters a robot body it has never seen? The answer determines whether cross-embodiment pretraining is a genuine shortcut or an expensive illusion.
Filling The Gaps The Early Releases Left
A dataset that records a million trajectories in one kitchen is not a large dataset; it is a very thorough diary of one room.
That tension between thoroughness in one setting and breadth across many is exactly the gap the next wave of datasets set out to fill. Three further datasets address gaps the earlier releases left open. RH20T (Robotic Hand 20 Tasks) collects human-teleoperated dexterous manipulation with synchronized force-torque and tactile sensor streams alongside RGB, giving policies access to contact information that camera-only datasets omit. RoboMIND (Multi-task Imitation from Natural Demonstrations) targets instruction-following generalization across 55 distinct tasks and provides dense language annotations at the sub-task level, making it well-suited for studying hierarchical language conditioning. AgiBot World, released by AgiBot in 2024, scales to over one million trajectories across more than 100 tasks on humanoid and mobile manipulator platforms, making it one of the largest single-organization releases and the primary benchmark for studying data-scaling behavior on dexterous humanoid bodies. To feel the scale: BridgeData V2 ships roughly 60,000 demonstrations; AgiBot World ships more than 1,000,000. Across scaling ablations published in 2024 (AgiBot World, 2024), policies trained on the smaller corpus typically plateau in success rate around 60-65% on held-out tasks, while the same architecture retrained on the larger corpus crosses 80% on those same tasks without any change in algorithm.
A common assumption is that the dataset with the most trajectories is always the best starting point, reasoning that more data means a stronger policy. In embodied AI this is wrong because trajectory count measures repetition, not coverage: a million trajectories recorded on one robot arm in one kitchen add almost no new generalization signal beyond the first hundred thousand, while sixty thousand trajectories spread across twenty-two robot bodies in twenty-one scene types provide qualitatively different inductive pressure. The correct mental model is that robot datasets are multidimensional: embodiment diversity, scene diversity, task diversity, annotation richness, and split design are each independent axes, and a dataset that scores high on only one axis will produce a policy that generalizes along only that axis, regardless of how large the trajectory count becomes.
| Dataset Question | Why It Matters | Example Evidence |
|---|---|---|
| Embodiments | Determines whether cross-robot transfer is being tested. | Robot type, arm count, gripper, base, sensors. |
| Scenes | Determines visual and physical diversity. | Homes, labs, kitchens, offices, institutions. |
| Tasks | Determines skill coverage. | Pick, place, open, wipe, pour, tool use. |
| Annotations | Determines what policies can condition on. | Language, goal images, success, failure, interventions. |
| Splits | Determines what generalization means. | Held-out tasks, held-out scenes, held-out objects, held-out robots. |
A dataset with fewer trajectories can be more valuable for a specific research question if it covers the held-out factor that deployment changes. Size, diversity, annotation richness, and split design are separate axes.
Use project repositories and Hugging Face dataset loaders before writing custom download code. The maintained loaders preserve released split names, metadata conventions, and feature schemas that custom scripts often flatten away.
The snippet below encodes a small dataset card and computes a crude coverage score, the same trade-off the scatter plot above showed visually, now turned into a single sortable number per dataset. It is not a substitute for careful evaluation, but it forces the reader to separate trajectory count from embodiment and task diversity.
# Compare dataset cards by coverage factors, not only trajectory count.
# The score is an analytical proxy for reading dataset claims critically.
datasets = {
"Open X-Embodiment": {"robots": 22, "tasks": 527, "scenes": 21},
"DROID": {"robots": 1, "tasks": 86, "scenes": 564},
"BridgeData V2": {"robots": 1, "tasks": 24, "scenes": 24},
}
for name, card in datasets.items():
coverage_proxy = card["robots"] * card["tasks"] * card["scenes"]
print(name, coverage_proxy)Step-Through: Computing The Coverage Proxy
Trace the coverage proxy with the three cards above, multiplying robots x tasks x scenes for each. Open X-Embodiment: 22 x 527 x 21. First 22 x 527 = 11,594, then 11,594 x 21 = 243,474. DROID: 1 x 86 x 564. First 1 x 86 = 86, then 86 x 564 = 48,504. BridgeData V2: 1 x 24 x 24 = 576. Now notice what the ranking hides: DROID lands at 48,504 entirely from its 564 scenes (its single robot contributes a factor of 1), while BridgeData V2's tiny 576 comes from having both few scenes and one robot. If you instead deployed on a new robot body, you would re-score with robots weighted heavily, and Open X-Embodiment (factor 22) would dominate even more sharply while DROID and BridgeData V2 (factor 1 each) would collapse. The single product number is only meaningful once you know which factor your deployment gap actually stresses.
The proxy ranks Open X-Embodiment highest because it multiplies robots, tasks, and scenes, not because that corpus wins every research question. It simply exposes more cross-factor coverage under this proxy. DROID's high scene count better serves in-the-wild visual generalization, and BridgeData V2's task framing better serves language-conditioned tabletop studies.
BridgeData V2 scores 576 on the coverage proxy, which sounds modest until you remember that 576 unique (robot, task, scene) combinations still represents more physical experience than most humans accumulate in a lifetime of kitchen chores. The robot has simply seen fewer kitchens, not fewer hours.
Mechanism: What The Dataset Teaches
Once you can read a dataset card as a set of coverage axes, the next question is what each axis actually does to the learned policy. Each major dataset teaches a policy through a different pressure. Open X-Embodiment pressures the model to find representations that survive changes in robot body and task source. DROID pressures the model to survive scene, collector, and household variation. BridgeData V2 pressures the model to connect manipulation behavior with language or goal-conditioned task descriptions. RoboNet-style video-interaction datasets pressure predictive models to learn how robot actions change visual futures.
Think of each dataset as a chef trained in one culinary tradition. A chef who has cooked in twenty countries handles unfamiliar kitchens confidently, but may freeze when asked to adjust for altitude because altitude variation was never part of the training. A chef who has cooked thousands of recipes in one climate handles altitude perfectly but is lost in an unfamiliar kitchen layout. The gap that matters is always the axis that was held constant during training, and no single training corpus covers every axis at once.
This distinction matters when choosing pretraining data, because each dataset applies pressure along only one axis at a time: this is called the diversity-axis mismatch problem, and it is the main reason a policy can score well on one benchmark while failing silently on a different deployment gap. A model trained on many robots may learn robust visual affordances but still need an adapter for a new action space. A model trained in many homes may learn visual robustness but remain tied to one hardware platform. A model trained with rich language may follow instructions better while still failing on contact dynamics that the dataset under-sampled.
When a policy trained on a major dataset fails, ask which distribution did not transfer: visual scene, object category, task instruction, action representation, robot morphology, or operator style. This source-aware diagnosis is stronger than saying "the dataset was too small" because it points to the next data collection or adaptation step.
Consider a representative case reported in practice: an RT-X policy pretrained on Open X-Embodiment (which heavily represents tabletop scenes under controlled lighting) is deployed on a mobile manipulator in a home kitchen with under-cabinet shadows and a cluttered counter. In cases like this, grasp success rate can drop from roughly 70% in lab conditions to under 30% in deployment, but trajectory count and task diversity metrics gave no warning. The failure is not a scale problem; it is a scene-illumination and clutter gap that scene-count statistics do not surface. The correct fix is targeted DROID-style in-the-wild collection or domain randomization, not simply adding more tabletop data.
Toolchain And Split Advice
A practical researcher should load the official release first, preserve its native split names, then create a project-specific split manifest only after deciding what generalization claim is being tested. For Open X-Embodiment, that may be held-out robot or held-out task. For DROID, it may be held-out scene, collector, or object. For BridgeData V2, it may be held-out instruction, environment, or goal image. A split that ignores the dataset's main source of diversity wastes the reason to use that dataset.
When loading Open X-Embodiment via the tensorflow_datasets RLDS loader, pass data_dir pointing to your local mirror and set decoders={'steps': tfds.decode.SkipDecoding()} to avoid decoding every image frame eagerly; without this flag, a single dataset iteration can allocate several gigabytes of GPU-resident tensor memory before your training loop sees a single batch. Preserve the episode_metadata dict (which carries robot_type, file_path, and contributor fields) rather than dropping it during collation: these fields are the only way to reconstruct per-source accuracy breakdowns after training, and re-running a full dataset pass to recover them is expensive. If you need a quick sanity check, filter to dataset_name == 'bridge' first, since BridgeData episodes are smaller and will complete a dry run in minutes.
This is the reading-and-choosing procedure the Big Picture promised: apply it to any dataset card, including ones not covered in this section, to turn "which corpus is largest" into "which corpus matches my generalization claim."
- Name the deployment gap: robot, scene, object, task, language, or contact regime.
- Choose the dataset whose diversity axis matches that gap.
- Preserve official metadata and split fields during loading.
- Create one held-out split that changes the deployment gap explicitly.
- Report per-source results before reporting aggregate success.
Trajectory counts, task counts, scene counts, and language labels are often defined differently. Treat them as dataset descriptors, not as directly comparable leaderboard numbers unless one audit script normalizes them.
A researcher studying kitchen generalization may prefer DROID-style scene diversity. A researcher studying robot-body transfer may prefer Open X-Embodiment. A researcher studying language-conditioned tabletop manipulation may begin with BridgeData V2.
Real-World Application: Robot Foundation Models
Physical Intelligence's pi0 model is pretrained on a mixture dominated by Open X-Embodiment and BridgeData V2, then fine-tuned on task-specific data for behaviors like folding laundry and clearing tables. The cross-embodiment pooling is exactly what lets a single pi0 checkpoint drive multiple distinct robot arms, while the targeted fine-tune supplies the contact and scene specifics the pooled corpus under-samples. This is the diversity-axis reasoning from this section running in a deployed system.
Unified action tokenization across embodiments. The 2024-2025 wave of large robot foundation models (pi0 from Physical Intelligence, 2024; RoboVLMs, 2024) tokenizes continuous actions via flow-matching (where a network learns to transform random noise into a valid action trajectory through a sequence of small denoising steps, rather than picking from a fixed vocabulary of discrete bins) instead of discrete bins, preserving force resolution while sharing a single decoder across heterogeneous bodies. The open architectural question is whether a flow-matching head can generalize its denoising trajectory to an embodiment it has never seen during pretraining, without per-robot fine-tuning.
Automatic data quality filtering at dataset scale. AgiBot World (2024, AgiBot) and the LeRobot v3 dataset format both expose per-episode outcome labels and intervention flags, enabling quality-weighted training. Research in 2024-2025 shows that selecting the top 20% of trajectories by a learned reward model outperforms training on the full corpus, but the reward model itself requires a small labeled seed set. Scaling quality filtering to million-trajectory corpora without human re-labeling is an active direction at several labs.
Dataset mixture optimization as a first-class problem. Cross-embodiment policies trained on a fixed mixture of Open X-Embodiment sources show large variance depending on source weights (OpenVLA, 2024, Stanford and Berkeley). Learned mixture policies that adapt weights dynamically during training are being explored, but no method yet handles the combinatorial space of all released robot datasets simultaneously.
Open problem: Build a dataset-mixture oracle that predicts, from dataset card metadata alone (embodiment count, scene count, task count, annotation richness), the optimal mixing weight for a held-out target deployment, without running full pretraining for each candidate mixture. A tractable first step is a regression model trained on the existing published ablation tables from RT-X, OpenVLA, and pi0.
For each major dataset, can you name the held-out factor its evaluation most strongly tests? If not, you know the dataset name but not the scientific claim.
Major robot datasets are not interchangeable warehouses. Each encodes a choice about embodiment, scenes, tasks, annotations, and what kind of generalization deserves evidence.
Choose two robot datasets and write a two-row dataset card comparing robot bodies, scenes, tasks, labels, license, and split design.
Project Ideas
Beginner (weekend): Build a dataset coverage visualizer that loads BridgeData V2 episode metadata via the LeRobot Python library and plots the distribution of tasks, scenes, and trajectory lengths as interactive charts; the key challenge is that episode metadata fields are inconsistently populated across contributors, so you must write a robust normalization pass before any aggregation is meaningful. Intermediate (1-2 weeks): Train a BC policy on a 5,000-episode subset of Open X-Embodiment using LeRobot and MuJoCo for offline evaluation, then measure per-source success rates broken out by robot type; the key challenge is constructing a held-out split that isolates one robot body so the generalization gap is attributable to embodiment rather than task difficulty. Intermediate (1-2 weeks): Use ROS2 and a PyBullet simulated UR5 to collect 500 pick-and-place demonstrations with a keyboard teleoperation script, export them in RLDS format compatible with the Open X-Embodiment schema, and verify that a pretrained RT-X checkpoint can fine-tune on your data; the key challenge is aligning your custom action space and camera intrinsics with the contributor metadata fields expected by the shared RLDS loader.
Lab: Reading A Real Robot Dataset
Goal: Confirm empirically that trajectory count and diversity are independent axes by inspecting a real corpus rather than a synthetic card. Tools: Python with the lerobot library (pip install lerobot) and a Hugging Face account; no GPU required. Steps: Load a public LeRobot dataset such as lerobot/bridge via LeRobotDataset, then read its episode-level metadata. What to vary: Group episodes by task label and by scene or contributor field, then compute, for each group, the episode count and the mean episode length in steps. What to observe: Plot a histogram of episodes-per-task. You should see a heavy-tailed distribution where a handful of tasks hold most trajectories, so the total trajectory count is a poor predictor of how many distinct tasks or scenes the policy actually sees. Then recompute the coverage proxy from this section using the real distinct task and scene counts, and compare it against the raw trajectory total to feel the gap between repetition and coverage. Plan for 15 to 30 minutes including download.
What's Next
Section 24.2 turns those comparison questions into a concrete schema for dataset structure, metadata, licensing, and dataset cards.
Khazatsky, A. et al. (2024). DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset.
Provides an in-the-wild manipulation dataset with diverse scenes, collectors, tasks, and detailed hardware reproduction guidance.
The central reference for cross-embodiment robot data, standardized dataset release, and RT-X style transfer across robot bodies.
Walke, H. R. et al. (2023). BridgeData V2: A Dataset for Robot Learning at Scale.
A large manipulation dataset designed around open-vocabulary multi-task learning, goal images, language, and data-scale experiments.
Google DeepMind Open X-Embodiment Repository.
Shows the released dataset structure and RLDS episode organization used by the Open X-Embodiment ecosystem.
LeRobotDataset v3.0 Documentation.
The practical reference for standardized multimodal robot time-series data, metadata, indexing, and Hub visualization.