Section 37.2: Learning dynamics models; ensembles and uncertainty

A planner without a model is blind. A planner with one wrong model is confidently blind.

A Budget-Conscious MPC Loop

Ensembles And Predictive Distributions

Ask a single learned model what happens next and it will always answer with a number, even when the honest answer is "I have never seen ground like this"; the trick is to ask several models at once and listen for how loudly they disagree. To make that disagreement measurable, a standard robotics model predicts state deltas rather than absolute next state:

$$ \Delta \hat s_t = f_\theta(s_t, a_t), \qquad \hat s_{t+1} = s_t + \Delta \hat s_t. $$

Training deltas often improves conditioning. Ensembles then estimate epistemic uncertainty (uncertainty caused by insufficient training data, as opposed to irreducible sensor or process noise) by disagreement across several bootstrap models, where each member is trained on its own resample-with-replacement subset of the same transition buffer, so members only disagree where the data is thin (the exact resampling mechanism is walked through in detail below), as sketched in Figure 37.2A: the same query state branches into several plausible futures, and the width of that fan is the trust signal a planner reads. PETS (Probabilistic Ensembles with Trajectory Sampling) is the canonical reference for combining such ensembles with trajectory sampling in planning.

The diagram above traces this flow concretely. One input state fans out to four bootstrap members. Their individual deltas (0.09 to 0.15) collapse into a single mean of 0.113 plus a spread of 0.06. That spread, not the mean, is what the planner consumes as its trust signal. For embodied AI this matters because a physical robot cannot roll back a committed action. When a single model is confidently wrong, the robot has already moved. Ensemble disagreement gives the planner a real-time signal before commitment: high spread across members means the robot is near the edge of its training distribution, where the next footstep or grasp is mechanically risky rather than merely statistically uncertain. The practical payoff is striking. PILCO (Probabilistic Inference for Learning Control, a policy-search method that plans directly against a Gaussian-process dynamics model instead of a neural-network ensemble) uses a GP model that exposes uncertainty rather than suppressing it, and it learns a cartpole swing-up task in roughly 30 real-world trials. A model-free policy trained without any uncertainty signal typically needs thousands of trials to reach the same control quality. Every episode in the dark wastes a physical interaction that uncertainty-aware planning would have avoided or shortened.

The bootstrapping mechanism works as follows. Each ensemble member is trained on a resample with replacement from the same transition buffer, producing slightly different training sets. Members that generalize similarly to a new state give low spread; members that generalize differently give high spread. That spread typically tracks sensitivity to dataset variation, a proxy for how far the query state lies from well-covered training data (though, as the Warning callout below notes, agreement is not a guarantee of correctness).

Delta prediction reduces the dynamic range the learner must model. This matters most when positions or joint angles evolve smoothly step to step. Bootstrapped ensembles then measure how sensitive each prediction is to dataset variation. Pair those two choices with horizon-conditioned evaluation and the planner gains a clearer signal about both local fit and out-of-support risk.

Where Dynamics Learners Break

Making ignorance visible only helps if you also know the specific ways a dynamics learner turns confident and wrong, so it is worth cataloguing where these models actually fail.

Two failure modes are especially common. The first is representation collapse: the model input omits a latent variable that actually drives the transition, such as slip state, cable tension, or tool wear. In that case every ensemble member can agree and still be wrong. The second is train-test mismatch in rollout usage. You train the model one step at a time, then ask it to support five-step or ten-step ranking during planning. Small local biases then compound before the controller can correct them.

A model that cannot say "I do not know" will eventually say something confidently wrong at exactly the wrong moment.

Horizon Length as a Bounded Resource

Consider a specific case: a model predicts velocity with a consistent 2% upward bias per step. At step one the error is 0.02 m/s, negligible for a planner with a tight receding horizon (a control scheme that re-plans a short window of future actions at every step and only executes the first one). At step five the accumulated bias reaches roughly 0.10 m/s, enough to push a footstep plan past a stability margin. At step ten the drift is 0.20 m/s, at which point the planner is ranking trajectories against a fiction.

The one-step held-out loss looked acceptable throughout; only the rolled-out overlay reveals when the model stops being useful for control. This is why horizon length is not a free hyperparameter: it is bounded by the point where cumulative error exceeds correction bandwidth, and beyond that point the planner is ranking trajectories against a fiction.

Good failure analysis pairs one-step metrics with rolled-out overlays on a fixed panel. PyTorch Lightning, Weights & Biases, or plain structured JSON logs all work; the artifact just has to show where the predicted state first leaves the physically plausible band. That crossing is where the planner should have shortened its horizon, switched to a fallback controller, or asked for more data.

Worked Probe

Having seen why horizon length and hidden variables set the limits of trust, it helps to watch the trust signal itself emerge from concrete numbers.

The next probe predicts a one-step velocity delta from four ensemble members and logs both the mean transition and the disagreement the planner should read.

# Aggregate one-step delta predictions from a tiny ensemble.
members = [0.09, 0.10, 0.11, 0.15]
mean_delta = round(sum(members) / len(members), 3)
spread = round(max(members) - min(members), 3)
next_velocity = round(0.6 + mean_delta, 3)
print(
    {
        "delta_members": members,
        "mean_delta": mean_delta,
        "spread": spread,
        "predicted_next_velocity": next_velocity,
    }
)