Section 35.6: Data scale, compute, and the open-vs-closed divide

"Every scaling curve is secretly a budget memo with philosophical opinions about openness."

A Research Lead With Spreadsheets

Why Scaling In Robotics Is Different

Buy ten thousand more GPUs and a language model gets smarter overnight. Buy ten thousand more GPUs for a robot policy and the arm still has to physically pick up the cup ten thousand times before you learn anything. That is the central tension Figure 35.6A sets up. Data scale and compute are two of the three forces that drive embodied AI progress. The third one, real-world evidence, is the one most scaling stories undercount. Language models can often scale by adding text and compute while keeping the evaluation channel cheap. Robotics is more stubborn. Each extra demonstration may require hardware time, operators, resets, safety review, and embodiment-specific calibration. A compute-heavy training run can still stall on the cost of generating trustworthy robot data and validating it on real platforms. To make this concrete: a text corpus can grow by one billion tokens overnight via a web crawl. Adding one thousand new robot demonstrations requires booking hardware, scheduling operators, running resets, and reviewing safety logs, which typically takes several weeks for the same order-of-magnitude gain in coverage. To feel the asymmetry directly: a language model that needs 50,000 training examples to learn a new skill can often get by with 300 once cross-task transfer is working. But a robot policy trained on 300 grasping episodes from one arm cannot carry that count to a second arm with a different wrist camera; the team must re-collect most of those episodes from scratch.

This asymmetry sets the tempo of each stack. Closed systems post strong numbers on data and infrastructure no one else can touch. Open stacks trail on headline scores but iterate faster, because the community can debug, fine-tune, and independently validate every step.

A Simple Scaling Lens

If real-world evidence generation is the hidden expensive term, it helps to write down a compact expression that puts that term next to the two everyone already tracks.

A stylized way to think about the trade-off is

$$P \approx f(D, C, E),$$

where \(D\) is data diversity and volume, \(C\) is compute for training and serving, and \(E\) is evaluation throughput on trustworthy scenario panels. Many labs can buy more compute faster than they can buy more credible robot evidence, which means progress saturates on the least glamorous axis.

A more operational view is to treat research throughput as

$$T_{\mathrm{iteration}} \approx \max(T_{\mathrm{data\ prep}}, T_{\mathrm{training}}, T_{\mathrm{hardware\ evaluation}}) + T_{\mathrm{failure\ analysis}}.$$

This decomposition makes the open-versus-closed divide more concrete. A closed stack may lower apparent model-development time because the policy arrives pre-integrated, but it often raises attribution cost because the lab cannot inspect which part of the performance came from data curation, architecture, post-training, teleoperation quality, or evaluation filtering. An open stack may start from a weaker absolute capability level while still producing faster scientific learning because failures are easier to localize and adaptation loops are easier to rerun.

Code Fragment 1 turns this intuition into a toy budget calculation.

# Compare where the budget goes in two robot-foundation-model programs.
programs = {
    "open_lab": {"data_hours": 400, "gpu_days": 120, "real_eval_days": 40},
    "closed_frontier": {"data_hours": 5000, "gpu_days": 900, "real_eval_days": 120},
}

for name, vals in programs.items():
    evidence_pressure = vals["real_eval_days"] / vals["gpu_days"]
    print(f"{name}: evidence_pressure={evidence_pressure:.2f}")

open_lab: evidence_pressure=0.33
closed_frontier: evidence_pressure=0.13

The expected output is a higher evidence-pressure ratio for the smaller open program, meaning a larger fraction of its iteration budget is spent on real-world validation instead of pure training throughput. That does not automatically make the open path worse; it often means the lab is paying for more inspectable evidence per unit of model development.

Mechanisms Behind The Cost Curve

The budget lens tells us where the money goes, but to see why open and closed stacks respond so differently to that spending we have to look under the ratio at the mechanisms that turn data and compute into policy performance.

Data scale matters in at least three different ways: the number of embodiments represented, the diversity of tasks and scenes, and the quality of state-action alignment inside each trajectory. Compute interacts with those axes asymmetrically. More FLOPs can help fit larger mixtures or more expressive action models, but they cannot repair missing calibration metadata, poor reset discipline, or under-specified action semantics.

This is where the open and closed worlds diverge mechanistically. Open stacks usually expose dataset schema, action conventions, and training code, so their common failure mode is limited scale or uneven embodiment coverage. Closed stacks may demonstrate stronger integrated performance, but their common scientific weakness is that the source of the gain becomes entangled with private data mixtures, private evaluation filters, and hidden post-training procedures.

Open Versus Closed Is A Research Trade-Off

Consider a specific case: a lab wants to adapt a pretrained policy to a new tabletop manipulation task. If the base model is open (say, OpenVLA or SmolVLA on LeRobot), the team can inspect the action tokenization scheme. That scheme maps continuous joint angles or end-effector deltas into discrete vocabulary tokens the language backbone can consume. The team confirms that the new task's state space is covered, runs a LoRA fine-tune in a day on a single A100, and pinpoints regressions by diffing the checkpoint against the pretrained weights. Action tokenization matters physically because bin boundaries set the finest motion the policy can express. Coarse bins force jerky, quantization-clipped trajectories on joints that need sub-millimeter precision. Overly fine bins inflate the vocabulary and destabilize training on small datasets. The scheme discretizes each action dimension into \(K\) uniform or learned bins, appends those token ids to the language context, and trains the model to predict them autoregressively. The policy inherits the backbone's context-window and sampling machinery at the cost of treating motion as vocabulary. If the base model is closed, the team must use a vendor fine-tuning API, accept opaque action conventions, and read results only through the aggregate metrics the vendor returns. The open path takes longer to reach the vendor's peak capability, but the team knows exactly why each failure occurs and can fix it. This distinction matters most during the debugging phase, not the demo phase.

Evaluation Consequences

Because that debugging advantage lives entirely in what the team can inspect, the stack decision does not just change how fast a lab iterates; it changes what kind of scientific claim its evaluations can support at all.

The choice of stack changes what kind of science a lab can do. Closed systems such as GR00T N1.5 are useful as frontier capability references, but they are weak substrates for ablations that isolate physical causes: if a Franka Panda wrist camera mounted 2 cm off-nominal degrades grasp success from 87% to 61%, a closed stack cannot tell you whether the loss traces to the visual encoder, the action tokenizer, or a training-set bias toward cameras mounted flush with the end-effector plate. Open systems such as OpenVLA or SmolVLA on LeRobot expose the action tokenization scheme and the trajectory schema, so the team can diff the failing episode against the nearest training neighbor, check whether the off-nominal camera pose falls outside the \(\pm\)5 mm mounting-offset distribution recorded in the dataset manifest, and rerun a single LoRA fine-tune with augmented viewpoints rather than re-collecting thousands of demonstrations.

A model that scores well on a benchmark no one else can run is not a result; it is a private opinion about performance.

The minimum reproducible evidence bundle

The minimum evidence bundle for a reproducible scaling claim must include one construct-matched task panel run on physical hardware with embodiment labels (robot model, end-effector serial number, controller firmware version), dataset provenance (which Open X-Embodiment or LeRobot split was used), the training or fine-tuning config including LoRA rank and learning rate, per-joint latency notes (a 15 ms spike on the Franka wrist joint during high-torque grasps shifts the action distribution in ways that aggregate success-rate metrics hide), and a failure taxonomy keyed to physical cause (contact slip, sensor dropout, out-of-distribution lighting) saved in the same artifact. Without that bundle, a stronger demo may still be a weaker scientific claim.