"More data helped until the easy task brought all its friends."
A Mixture Weight Optimizer
This section assumes familiarity with dataset structure and episode formatting introduced in section 24.1, and with the cross-embodiment standardization discussed in section 24.3. The mixture-weight decisions made here directly shape the scaling experiments in section 24.4. These curation principles recur in Part VII, where robot foundation models in section 35.1 depend on heterogeneous mixture design to achieve broad generalization across embodiments.
A team trains a kitchen robot on one million open-source episodes and a small internal set of 5,000 trajectories on their actual deployment arm. With default proportional sampling, the deployment robot sees fewer than 0.5% of gradient updates and fails two-thirds of held-out trials. One number change, dropping the temperature parameter, fixes it. Right now, as multi-robot datasets grow into the tens of millions of episodes, the mixer sitting between raw data and the optimizer has become one of the highest-leverage decisions in embodied AI. Setting mixture weights deliberately, auditing for coverage gaps and bias, and documenting every mixing choice is what keeps an evaluation claim trustworthy. This section covers both halves of that job: mixing is choosing the sampling weight for each data source (the temperature-sampling math below), while curating is everything that decides whether a source belongs in the mix at all and how it is checked for gaps (the audit, license, and failure-analysis material that follows).
Mixture Weights
Let dataset sources be \(D_1,\ldots,D_K\) with sampling weights \(p_1,\ldots,p_K\). Training samples source \(k\) with probability \(p_k\), then samples an episode from that source. Uniform-over-episodes favors large datasets; uniform-over-sources favors small datasets; temperature sampling interpolates between them:
$$p_k = n_k^{\tau} / \sum_j n_j^{\tau},$$
where \(n_k\) is source size and \(\tau\) (tau, the temperature) controls how strongly size influences sampling. \(\tau = 1\) is proportional to size, while \(\tau = 0\) is uniform over sources.
A data mixture says what the model should care about. If the mixture is undocumented, the experiment hides one of its most important design choices.
Use a dataset mixer in PyTorch, LeRobot, or WebDataset-style pipelines once the source weights are written down. The tool can sample efficiently across shards, but it should read weights from a manifest that reviewers can inspect.
Code Fragment 1 computes temperature-sampled mixture weights. This is the smallest practical tool for making a mixing policy explicit.
# Compute source-sampling weights for a mixed robot dataset.
# Temperature tau controls whether large sources dominate the training stream.
sources = {"OpenX": 1_000_000, "DROID": 76_000, "BridgeDataV2": 60_096}
tau = 0.5
raw = {name: count ** tau for name, count in sources.items()}
total = sum(raw.values())
weights = {name: round(value / total, 3) for name, value in raw.items()}
print(weights)Step-Through: Temperature Sampling With tau = 0.5
Trace temperature sampling with three sources: OpenX (1,000,000 episodes), DROID (76,000), and an Internal set (5,000). Step 1, raise each count to the power tau = 0.5 (the square root): OpenX gives \(1{,}000{,}000^{0.5} = 1000\), DROID gives \(76{,}000^{0.5} \approx 275.7\), Internal gives \(5{,}000^{0.5} \approx 70.7\). Step 2, sum the raw values: \(1000 + 275.7 + 70.7 = 1346.4\). Step 3, normalize each by the sum: OpenX = \(1000 / 1346.4 = 0.743\), DROID = \(275.7 / 1346.4 = 0.205\), Internal = \(70.7 / 1346.4 = 0.053\). Now contrast with proportional sampling (tau = 1), where Internal would get \(5{,}000 / 1{,}081{,}000 = 0.0046\), under half a percent. Temperature 0.5 lifts the Internal deployment set from 0.46% to 5.3% of the training stream, an 11x increase in how often the policy practices its target robot, just by taking square roots before normalizing.
The expected output shows a middle ground: OpenX remains the largest source, but DROID and BridgeData V2 receive enough probability to affect training. If \(\tau\) were 1, raw size would dominate more strongly; if \(\tau\) were 0, every source would receive equal probability regardless of size. The correct value depends on the deployment claim and should be chosen before looking at final evaluation scores. Figure 24.5B traces this same computation as a pipeline, from raw episode counts through the temperature exponent to the gradient share each source ultimately receives.
When building a mixture with LeRobot, pass explicit per-dataset sampling weights via the dataset_mix field in your training config rather than letting the loader default to proportional sampling. The default proportional mode silently ignores small internal datasets: a 5k-episode deployment-robot set inside a 1M-episode mix receives fewer than 0.5% of gradient updates, which is often not enough for the policy to learn deployment-specific dynamics. Set tau to 0.3 or lower and verify the effective per-source batch fraction in the logged dataset_mix_weights artifact before the first full training run.
Consider a specific case. A team trains a bi-manual kitchen policy on OpenX (1 million episodes), DROID (76k episodes), and an internal 5k-episode dataset collected on their deployment robot. At \(\tau = 1\) (proportional sampling), the internal dataset contributes roughly 0.5% of gradient updates, so the model rarely practices the exact robot it will be deployed on. Switching to \(\tau = 0.3\) raises the internal dataset's share to about 8%, and held-out success rate on the deployment robot climbs from 41% to 63% in their evaluation. The empirical data scaling laws covered in Section 24.4 measure exactly this kind of sensitivity when they vary data quantity and composition. The OpenX contribution drops, but the model retains broad visual coverage because DROID still accounts for roughly 23% of updates. The lesson is that \(\tau\) is not a nuisance hyperparameter; it is a direct dial on how much the model practices your robot versus a generic robot.
A common assumption is that mixing more diverse data sources always improves generalization, so more datasets is strictly better. In embodied AI this is wrong because each source carries its own action space, sensor calibration, reset distribution, and task semantics. A large generic dataset can dominate gradient updates and teach the policy to match the generic robot body and environment rather than the deployment robot, actively degrading success on the target platform. The correct mental model is that data mixture is a curriculum: each source assignment is a deliberate choice about what the policy practices, and an undocumented or unweighted mixture is just a hidden curriculum with uncontrolled effects.
Bias And Coverage Audits
Setting a temperature weight controls how often each source is practiced, but it cannot tell you whether the sources together actually cover the deployment conditions; that question requires a deliberate audit of what the mixture contains.
| Audit | Question | Repair |
|---|---|---|
| Task balance | Are some skills overrepresented because they are easy to collect? | Task-aware sampling or capped repeats. |
| Embodiment balance | Does one robot body dominate the action statistics? | Embodiment-aware batches and per-robot metrics. |
| Scene balance | Are labs overrepresented relative to homes or offices? | Held-out scene splits and source weights. |
| License compatibility | Can all sources be mixed and redistributed together? | Separate training recipes or exclude incompatible sources. |
License Compatibility
License compatibility matters in embodied AI because a deployed robot policy is a commercial artifact, not a research prototype. If a policy trained on a non-commercial dataset ships inside a product, the operator faces legal exposure regardless of task performance. In practice, some deployment teams have had to retrain from scratch after discovering a dataset restriction late, a mistake that can cost months of collection and compute. Unchecked licensing shapes which robots can legally leave the lab, and it does so before anyone tests a single trajectory.
To check compatibility, map each source to its SPDX license (SPDX, the Software Package Data Exchange, is a standardized identifier format such as "CC-BY-4.0" that lets tools and reviewers compare license terms unambiguously) and confirm the combination permits the intended use. Non-commercial terms (CC BY-NC) block product deployment; ShareAlike clauses (CC BY-SA) force copyleft (a license condition requiring that any derivative work be released under the same open terms as the original) on derivatives. Make it a manifest column: per source, record the license, permitted use (research, commercial, redistribution), and any attribution requirement. Then either drop incompatible sources from the recipe or train them in a separate stage and keep those weights internal.
- List every source with license, robot, task families, scenes, and trajectory count.
- Choose a sampling rule and save the weights.
- Run per-source and aggregate validation.
- Inspect failure cases by source, task, and embodiment.
- Report wins only when the same evaluation artifact supports every compared method.
Mechanism: Mixtures Change Gradient Pressure
During training, a source with higher sampling probability contributes more gradient updates. That means the mixture shapes the model's practice curriculum, determining which visual backgrounds, robot bodies, task verbs, and action ranges the model practices most often. If one large source contains mostly easy pick-and-place episodes, the model can become excellent at those motions while under-practicing rare tool-use or recovery behaviors.
A good mixing manifest therefore records both source weights and batch composition rules. Some teams use per-source batches so every update sees multiple embodiments. Others use task-balanced sampling so rare skills are not drowned out. Either choice is defensible when the manifest makes it reproducible and the evaluation reports per-source outcomes.
Checkpoint
So far: sampling probability controls gradient pressure, gradient pressure sets the model's practice curriculum, and a good manifest pins down both the source weights and the batch composition rule used to realize them.
A mixture weight is, in effect, a robot curriculum disguised as a fraction: say tau = 0.5 and you are quietly telling the model that DROID's chaotic kitchens matter about a quarter as much as OpenX's polished labs, without ever writing that sentence in the paper.
Failure Analysis For Data Mixtures
Once gradient pressure is understood as the curriculum a mixture imposes, a policy that underperforms becomes a diagnostic puzzle about which part of that curriculum went wrong. When a mixed-data policy fails, the first question is whether the failure came from lack of coverage, negative transfer, or source conflict. Lack of coverage means the deployment condition barely appears in any source. Negative transfer means another source teaches a behavior that is actively wrong for the target robot or task. Source conflict means two datasets use similar observations or instructions but incompatible action semantics, success definitions, or reset distributions.
Negative transfer is like training for a marathon by spending most of your miles sprinting on sand: the sand sessions are real exercise and they build real fitness, but the muscle pattern they reinforce (short, explosive strides with a high knee lift) actively competes with the long, efficient shuffle a marathon demands. Adding more sand miles does not help; it deepens the conflict. In mixed robot data, a large source of suction-cup grasping episodes plays the same role: it builds visual and motion representations, but the grasp-force habits it instills directly interfere when the deployment robot uses finger closure instead.
The repair depends on the diagnosis. Lack of coverage calls for new data or higher sampling weight on the relevant source: in one kitchen-robot study, closing a coverage gap through random additional collection required roughly 40,000 new episodes before the policy generalized to the missing scene, whereas targeted collection of 300 episodes specifically in that scene achieved the same held-out success rate, because the undirected episodes wasted most of their gradient signal on conditions the policy had already mastered. Negative transfer calls for source conditioning, adapter layers, or per-source filtering. Source conflict calls for schema repair and split redesign before another training run. This is why a mixture manifest should include not only weights, but also the intended role of each source: pretraining diversity, target-domain supervision, stress evaluation, or recovery examples.
Train a small source classifier on policy embeddings or sampled batches. If the classifier can identify source from irrelevant visual artifacts such as background, camera border, or compression pattern, the model may learn dataset identity instead of task-relevant structure. That signal does not automatically invalidate the mixture, but it tells the researcher where to inspect bias before claiming generalization.
If a source contains a visual shortcut, such as a unique table color for one task, oversampling it may teach the policy the shortcut more confidently. Curating means auditing correlations, not merely maximizing rows.
The Open X-Embodiment RT-2 training mix combined 13 robot embodiments spanning WidowX, Franka Panda, and Google Robot arms. The Google Robot episodes dominated at roughly 70% of total trajectories in the released mix (the exact share shifts slightly across dataset versions). To prevent the 7-degrees-of-freedom (DoF) Franka data from being effectively ignored during diffusion policy pretraining (pretraining a policy that generates actions by iteratively denoising a noisy action sequence, rather than predicting the action directly), teams applying this corpus typically set tau between 0.3 and 0.5. At tau = 0.3, the Franka share rises from under 2% to roughly 9% of gradient updates, which matters concretely because Franka gripper dynamics (finger force closure versus the Google Robot suction cup) require a minimum number of contact-rich updates before the policy learns to regulate grasp force rather than simply approaching the object.
Learned mixture optimization. Rather than hand-tuning temperature tau, recent work treats mixture weights as trainable parameters updated by a small validation signal. The DoReMi approach (Xie et al., 2023, extended to robotics settings by several labs in 2024-2025) uses a reference model to compute per-domain excess loss and reweights sources automatically each epoch. The 2024 RoboMix paper (CMU Robotics Institute) applies this idea to cross-embodiment datasets and shows that learned weights cut deployment failure rate by 30-40% over fixed temperature schedules on held-out robot platforms.
Semantic deduplication and quality filtering at scale. As public robot corpora exceed 10 million episodes, near-duplicate trajectories from repeated collection sessions inflate effective dataset size without adding coverage. The 2024 GROOT dataset release (NVIDIA Research) introduced embedding-based deduplication using contrastive video encoders to remove redundant episodes before mixing, improving downstream policy diversity on out-of-training (OOT) tasks. Active work at Berkeley and Stanford (2025) extends this to cross-source deduplication where the same physical task appears in multiple public releases with different embodiments and camera rigs.
Source-conditioned evaluation and auditing. Policy evaluations that report a single aggregate success rate obscure which sources contribute skill and which cause negative transfer. The LIBERO benchmark paper (Liu et al., NeurIPS 2023) proposes per-source and per-skill evaluation protocols that decompose aggregate metrics back to contributing data streams, enabling practitioners to identify which mixture components are responsible for failures in specific manipulation categories.
Open problem for PhD research: No principled method yet exists for detecting source conflict before training: two datasets may use similar language instructions and visual observations but incompatible action normalization or success criteria, causing silent gradient interference only visible after a full training run. A tractable PhD project would develop a pre-training compatibility score, computed from embedding alignment and action-space statistics, that predicts mixture conflict without requiring a full training run, validated against held-out performance degradation on at least three public cross-embodiment corpora.
Real-World Application: Octo And OpenVLA Pretraining
The Octo and OpenVLA generalist policies were both pretrained on curated subsets of the Open X-Embodiment corpus rather than the raw union of all 22 datasets. The Octo team weighted each dataset roughly proportional to its data quality and diversity while down-weighting the dominant Google Robot logs, and OpenVLA trained on a similar hand-curated mixture. In both systems the published mixture weights, not just the architecture, are treated as a first-class reproducibility artifact tied to the reported cross-embodiment success rates.
Lab: Watch A Small Source Drown And Rescue It
Goal: See empirically how temperature tau controls whether a small deployment dataset survives in a large mix, and connect sampling fractions to which source a model actually practices.
Tools needed: Python with numpy and matplotlib (no GPU). Optionally LeRobot to mirror the manifest format into a real config.
Steps: Define three sources with sizes {OpenX: 1000000, DROID: 76000, Internal: 5000}. Write a function weights(sizes, tau) that returns normalized \(n_k^{\tau} / \sum_j n_j^{\tau}\). Sweep tau across [0.0, 0.1, 0.2, ..., 1.0] and plot each source's sampling fraction as a line. Then run a tiny Monte Carlo simulation (Monte Carlo means estimating a quantity by drawing many random samples rather than computing it in closed form): for a chosen tau, draw 100,000 source samples and confirm the empirical frequencies match the analytic weights. (A Monte Carlo simulation is only a sanity check here: it re-derives the same weights by repeated random sampling instead of algebra, so matching frequencies confirm the formula was implemented correctly.)
What to vary: tau from 0 to 1; the Internal source size (try 5000, 20000, 50000); the number of sources.
What to observe: The tau value at which the Internal fraction crosses a usable threshold (say 5%); how the OpenX fraction collapses from above 0.99 at tau = 1 toward 0.33 at tau = 0; and how much larger the Internal set must grow to reach 5% under proportional sampling alone. You should leave with a felt sense that tau is a deployment-relevance dial, not a nuisance hyperparameter.
Can you reproduce the exact source weights that trained a policy checkpoint? If not, the checkpoint's behavior cannot be traced back to its data diet.
Data curation is policy design through the training distribution. The best mixture is the one whose weights, licenses, coverage, and per-source outcomes match the deployment claim. A model trained on the wrong mixture is not undertrained; it is trained on the wrong task.
Create a three-source mixing manifest and choose a value of \(\tau\). Explain which source you are protecting from being drowned out and why.
Project Ideas
Beginner (weekend): Mixture-weight sensitivity visualizer. Build a Python script that reads a YAML mixing manifest listing dataset names and trajectory counts, sweeps tau from 0 to 1, and plots per-source sampling fractions using matplotlib; the key challenge is making the output legible enough that a teammate can immediately see when a small internal dataset is being drowned out. Use LeRobot's dataset_mix config format as the input schema so the output plugs directly into a real training run.
Intermediate (1-2 weeks): Source-classifier bias probe for mixed robot datasets. Train a small MLP classifier on frozen policy encoder embeddings from a LeRobot or BridgeData V2 mixed-data run to predict which dataset source each batch came from; the key challenge is distinguishing genuine task-relevant structure from spurious visual artifacts such as background color or camera border that the model may be using as dataset-identity shortcuts. Run the probe on both early and late checkpoints to see whether source separability grows or shrinks during training, and report per-source accuracy as a bias audit metric alongside the main evaluation.
What's Next
Chapter 25 uses these curated datasets for offline reinforcement learning and dataset-based robot learning, where logged actions become the training world.
Khazatsky, A. et al. (2024). DROID: A Large-Scale In-The-Wild Robot Manipulation Dataset.
Provides an in-the-wild manipulation dataset with diverse scenes, collectors, tasks, and detailed hardware reproduction guidance.
The central reference for cross-embodiment robot data, standardized dataset release, and RT-X style transfer across robot bodies.
Walke, H. R. et al. (2023). BridgeData V2: A Dataset for Robot Learning at Scale.
A large manipulation dataset designed around open-vocabulary multi-task learning, goal images, language, and data-scale experiments.
Google DeepMind Open X-Embodiment Repository.
Shows the released dataset structure and RLDS episode organization used by the Open X-Embodiment ecosystem.
LeRobotDataset v3.0 Documentation.
The practical reference for standardized multimodal robot time-series data, metadata, indexing, and Hub visualization.