Section 24.4: Empirical data scaling laws in imitation learning

"The curve kept improving, but only after we stopped mixing validation splits like smoothie ingredients."

A Careful Curve Fitter

Power-Law Habit

A common empirical form is:

$$E(N) = A N^{-\alpha} + E_{\infty},$$

where \(E(N)\) is an error or failure rate after training on \(N\) demonstrations, \(\alpha\) is the scaling exponent, and \(E_{\infty}\) is the irreducible floor under the current setup. The RT-1 and RT-2 evaluations at Google DeepMind and the DROID benchmark from Stanford and UC Berkeley report 1-minus-success-rate rather than raw success. This per-rollout failure fraction over a fixed task panel has a natural decreasing trend that fits the power law cleanly. This formulation assumes a well-controlled imitation learning setup where the data distribution holds steady between teleop collection and on-robot evaluation. Recalibrating a Franka Panda or nudging a wrist camera between training-data capture and the rollout sweep violates exactly that condition.

Why the Floor Never Reaches Zero

The irreducible floor \(E_{\infty}\) matters in embodied AI because physical systems have hard failure modes that data volume cannot fix. Joint backlash, calibration drift, sensor noise, and operator-induced bias in teleoperation (teleop) demonstrations all create a residual failure rate that persists regardless of dataset size. A policy trained on one million trajectories still fails when the real gripper's compliant finger deforms differently from every demonstration it saw. Ignoring \(E_{\infty}\) leads teams to overspend on data collection when the actual bottleneck is hardware or action representation. Figure 24.4B below sketches this shape: a steep early drop in error followed by a plateau at the irreducible floor.

Mechanically, the floor emerges because the imitation objective minimizes average loss over the collected distribution. Tasks or object poses near the tail of that distribution appear so rarely that even large datasets leave them underrepresented. The policy converges to low average error, but performance on rare configurations plateaus. Fitting the three-parameter model rather than a pure log-log regression (a linear fit between the logarithm of demonstration count and the logarithm of error, which recovers the scaling exponent as its slope) separates this structural ceiling from genuine data-driven improvement.

Code Fragment 1 fits a line in log-log space to estimate a rough scaling exponent. The example uses small synthetic numbers so the arithmetic is transparent.

# Estimate a rough imitation-learning scaling exponent from matched runs.
# Every point must come from the same robot, split, metric, and training recipe.
import math

demo_counts = [100, 300, 1000, 3000]
failure_rates = [0.42, 0.30, 0.20, 0.14]

x = [math.log(n) for n in demo_counts]
y = [math.log(e) for e in failure_rates]
x_bar = sum(x) / len(x)
y_bar = sum(y) / len(y)
slope = sum((xi - x_bar) * (yi - y_bar) for xi, yi in zip(x, y)) / sum((xi - x_bar) ** 2 for xi in x)
alpha = -slope
print("alpha:", round(alpha, 2))

Read the output alpha: 0.32 as a local empirical summary, not a universal robotics constant. Under this exact synthetic panel, each multiplicative increase in demonstrations cuts failure at a rate matching a log-log slope near 0.32. A real paper owes more: confidence intervals, seeds, task-level scatter, and whether the fit survives dropping the largest or smallest point.

Those caveats all reduce to a single demand: the exponent only means something when every point on the curve was produced under identical conditions, which forces a precise account of what each point actually represents.

What A Scaling Point Must Contain

A single point on a robot scaling curve is a bundle of choices: dataset subset, task panel, model capacity, action representation, training compute, evaluation seed policy, and success metric. If any of those choices changes between points, the plot may still describe a useful engineering trend, but it no longer isolates data scale. A scaling curve built without a locked, construct-matched evaluation panel is not a measurement of data value; it is a measurement of experimental drift. Serious scaling studies therefore save one artifact per point with the subset manifest, model config, training logs, evaluation videos, and per-task outcomes.

There are two common variants. A data-only scaling study fixes model class and grows the number or diversity of demonstrations. A joint scaling study grows data and model capacity together. Both can be valid, but they answer different questions and should not be described with the same claim.

To make this concrete: under a carefully controlled protocol, tripling the dataset from 1,000 to 3,000 demonstrations at exponent 0.5 cuts failure from 20% to roughly 12%. At exponent 0.2, the same tripling moves failure from 20% to only 17%. That gap determines whether a 10x data collection campaign justifies its cost. Published robot imitation-learning studies that control for split and metric report exponents in roughly the 0.2 to 0.5 range for table-top manipulation, consistent with the synthetic 0.32 in Code Fragment 1. Exponents near 0.5 appear in low-diversity settings where demonstrations fill a narrow distribution gap quickly. Exponents near 0.2 appear when the task panel is broad and new demonstrations cover increasingly rare edge cases. The difference is stark: cutting failure in half from a 20% baseline needs roughly 300 more demonstrations at exponent 0.5, but more than 50,000 at exponent 0.2. The same collection budget that solves a narrow task leaves a broad generalist policy almost unchanged. The Open X-Embodiment paper found that pooling across embodiments shifted the effective exponent. Cross-embodiment data added task coverage faster than it added per-task demonstrations, showing that the unit being counted changes the shape of the curve.