Section 37.5: Sample-efficiency advantages and failure modes

The most persuasive sample-efficiency plot is the one that survives contact with failure analysis.

A Budget-Conscious MPC Loop

Where The Efficiency Comes From

Squeeze one real robot trial for everything it is worth and it can teach your policy a dozen times over, which is the entire trick behind model-based sample efficiency: methods reuse real experience by either planning through a learned model or generating synthetic targets from imagination rollouts, so one transition can contribute to multiple policy-improvement updates rather than being consumed only once. The practical difference is striking: a model-free agent learning a manipulation task may need 2,000 real robot trials to reach acceptable performance, while a model-based counterpart reaches the same threshold in 300. That is the efficiency story as introduced in the model-free versus model-based trade-off analysis. Figure 37.5B plots both learning curves on the same axes so the horizontal gap between them, the sample-efficiency advantage, is visible directly.

But the same mechanism creates new failure terms:

$$ \text{deployment risk} \approx \text{model bias} + \text{optimizer error} + \text{timing overrun} + \text{uncertainty misuse}. $$

A serious evaluation must report both sample efficiency and this risk ledger on the same matched panel. Each of the four terms above gets its own treatment below: model bias first because it compounds the others, then optimizer error and timing overrun in the failure-mode table, and uncertainty misuse where it connects back to section 36.4.

This is where many benchmark stories become misleading. A method can hit target return quickly because the benchmark rewards an easy strategy that never exposes the hard parts of the dynamics, such as sparse collisions, rare slips, or recovery after contact surprises (cases where the real world departs from the conditions the model was trained on, often called distribution shift). If the evaluation panel does not include those cases, the sample-efficiency gain is real but incomplete. For embodied systems, incomplete often means unsafe.

What happens when the robot confidently follows its world model into a physical state it has never encountered? That is the everyday experience of a planner working at the edge of its training distribution. A model that was never wrong inside training has no way to announce when it starts being wrong outside it.

Why Model Bias Compounds

Model bias deserves particular attention because it silently multiplies every other entry in the table. A learned dynamics model only fits the transitions the robot has seen; it does not represent physical law. When the planner queries state-space regions the robot never visited during training, the model extrapolates. Extrapolation in high-dimensional dynamics is frequently wrong in physically meaningful ways: predicted torques violate joint limits, predicted contacts omit friction, and predicted slips ignore surface texture. A robot acting on those predictions can apply damaging forces, overbalance, or execute motions that are kinematically infeasible.

The mechanism is gradient-driven compounding. During planning, the optimizer finds action sequences that minimize predicted cost under the model. If the model assigns low cost to a trajectory through an unvisited region, nothing inside the planner stops it from selecting that trajectory. Each planning step shifts the robot slightly into lower-data territory, where model error grows, so the next plan starts from an even less-covered state. Real robots experience this as progressive drift. The first few steps look reasonable, then behavior degrades abruptly once the robot crosses the implicit boundary of its training distribution. In practice, a higher-capacity planner with a longer horizon tends to drift further and faster than a shallow one, because its optimizer searches a larger space of candidate trajectories and is correspondingly more likely to find a low-cost corridor that threads deep into unvisited state space before the model's errors grow large enough to trigger any internal alarm.

The table below catalogues the failure modes that recur across model-based deployments, pairing each with its typical symptom and the diagnostic artifact you can use to detect it.

Worked Probe

Those published budgets only become trustworthy once you can reproduce them in a single reportable artifact, so the natural next step is to package efficiency and its failure terms together. The next code fragment prints a compact evidence card for one benchmark comparison. This is the minimum artifact that should accompany a "sample efficient" claim.

# Build one evidence card for a sample-efficiency claim.
from dataclasses import asdict, dataclass

@dataclass
class EvidenceCard:
    target_return: float
    real_episodes_to_target: int
    planner_ms: int
    heldout_rollout_error: float
    dominant_failure: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

card = EvidenceCard(
    target_return=0.80,
    real_episodes_to_target=6,
    planner_ms=18,
    heldout_rollout_error=0.041,
    dominant_failure="model_bias_under_contact_shift",
)
print(card.as_row())

How To Audit The Claim

A convincing audit compares model-based and model-free baselines on the same reset panel, with the same observation contract, same seed count, and same target-return threshold. Then it adds deployment-facing fields that most papers omit: control period, average planner milliseconds, percentage of aborted rollouts, and the first failure mode observed during shift. That single table often explains more than several reward curves.

Even a table that captures every deployment field still measures only what happens at run time, and one large cost sits entirely off that ledger. Readers should also separate efficiency from engineering burden. A method that uses fewer real episodes but requires days of model tuning, fragile horizon schedules, and constant calibration babysitting is suffering from the hidden cost of model maintenance, and that cost may still make it the right choice for scarce-data robotics, but the trade should be stated explicitly.