Section 48.5: End-to-end and world-model driving

A world model lets the car ask "what happens if I do this?" without leaving the lane to find out.

On learned simulators for driving

A hard-braking cut-in at night in rain happens on the order of once every 200,000 km of real driving by rough industry estimate, yet a car that could dream would never have to wait for one: it could conjure ten thousand of them before sunrise. That is the world-model contract, and it asks a single question. Given a history of observations and a proposed action sequence, can we predict future observations (frames, occupancy, or flow) well enough that planning decisions made in imagination transfer to the real road? A well-calibrated world model can in principle synthesize those 10,000 rare scenarios overnight at a small fraction of the marginal cost of fleet driving (GPU time, not literally zero). Without a world model, a reliable detector would need very roughly 2 billion km of instrumented driving to encounter enough of these events at the same rate; with one, a single overnight GPU run can approximate, though not guarantee, the same statistical coverage, provided the generated scenarios are representative of the real long-tail distribution. As Figure 48.5A illustrates, the planner rolls the scene forward in the model's imagination and tests maneuvers against predicted futures before committing a single real command. The recurring question is closed-loop validity: a model can produce beautiful video yet still fail the planner if its predicted dynamics drift from real physics over the horizon. A world model that cannot transfer its judgments from imagination to asphalt is not a simulator; it is a storyteller.

Theory

From prediction to imagination

A learned world model approximates the environment transition \(p(o_{t+1} \mid o_{\le t}, a_t)\). Unrolling it produces an imagined rollout the planner can score, turning planning into search over action sequences inside a differentiable or sampleable simulator (differentiable meaning gradients can be backpropagated through the rollout; sampleable meaning the model can instead be queried by drawing samples, without needing gradients at all). This is the driving instance of the model-based control idea: plan in a learned model, act in the world.

Turning that abstract transition model into a driving simulator means committing to a concrete way of representing the future, and the three flagship systems below stake out the pixel-space end of that design space.

Generative driving world models

• GAIA-1 casts driving generation as next-token prediction over discretized video, text, and action tokens, producing controllable future driving videos.
• UniSim aims for a general interactive simulator: condition on an action and render the resulting next observation, so an agent can be trained and tested entirely inside the learned simulator.
• DriveDreamer conditions video diffusion on structured driving signals (HD map, 3D boxes, actions) to generate diverse, controllable scenarios for training perception and planning.

Occupancy flow

Rather than render pixels, occupancy flow predicts a future occupancy grid plus a flow field describing how occupied cells move. It is cheaper, directly consumable by planning (the planner wants to know what space will be blocked), and avoids hallucinating photorealistic but dynamically wrong frames.

This matters in embodied AI because a physical vehicle has hard kinematic limits: it cannot teleport, stop instantly, or pass through another object. A pixel-level video generator can satisfy perceptual metrics while violating these constraints silently, producing frames where a car appears to slow but the implied closing speed remains dangerous. Occupancy flow encodes exactly the quantities the actuator layer needs: which cells are blocked and how fast they are moving, giving the planner a collision check that is grounded in real physical geometry rather than appearance.

How the network is trained and queried

Mechanically, the network predicts two tensors over a fixed bird's-eye-view grid at each future timestep: a binary (or probabilistic) occupancy map and a 2-D velocity vector per occupied cell. To train the flow vectors, the model warps the current occupancy forward with the predicted velocities and minimizes a Chamfer or cross-entropy loss (where the Chamfer loss is the average distance from each predicted occupied point to its nearest ground-truth occupied point, and vice versa, penalizing both missed and spurious occupied cells) against the ground-truth future occupancy. At inference the planner does two things. It convolves the ego vehicle's footprint against the predicted occupied cells to compute collision probability, and it integrates the flow vectors to project cell positions further ahead without running additional network passes.

Figure 48.5B below traces this closed-loop planning cycle as a five-stage diagram, from observing the current state through imagining candidate futures to executing the chosen action.

Worked Example

The example strips the planning loop to its core. A toy action-conditioned predictor unrolls each candidate acceleration, and the planner keeps the action whose imagined future holds the largest safety margin, mirroring the real closed-loop pattern without a learned network.

import numpy as np

# Toy learned world model: predicts future ego-lead gap given an ego action.
# In reality this is a neural net rolling out future frames or occupancy.
def world_model_rollout(gap0, ego_v0, lead_v, accel, horizon=3.0, dt=0.1):
    """Imagine the gap trajectory under a constant ego acceleration."""
    gap, ego_v, gaps = gap0, ego_v0, []
    for _ in range(int(horizon / dt)):
        ego_v = max(0.0, ego_v + accel * dt)
        gap += (lead_v - ego_v) * dt          # lead minus ego closing rate
        gaps.append(gap)
    return np.array(gaps)

gap0, ego_v0, lead_v = 18.0, 14.0, 9.0        # closing on a slower lead
candidates = [-3.0, -2.0, -1.0, 0.0, 1.0]     # candidate ego accelerations

best_a, best_score = None, -np.inf
for a in candidates:
    future = world_model_rollout(gap0, ego_v0, lead_v, a)
    min_gap = future.min()                     # imagined worst-case safety margin
    # Reward keeping a margin while not braking harder than needed.
    score = min_gap - 0.5 * abs(a)
    print(f"accel={a:+.1f}  imagined_min_gap={min_gap:5.1f} m  score={score:5.2f}")
    if min_gap > 3.0 and score > best_score:    # require a hard safety floor
        best_a, best_score = a, score

print("chosen action (m/s^2):", best_a)

Expected output: aggressive positive accelerations let the imagined gap fall below the 3 m floor and are rejected; a mild deceleration wins because it preserves the safety margin at the least control cost. Swapping in a real GAIA-1- or UniSim-style model replaces world_model_rollout with a learned rollout while keeping this select-by-imagined-margin structure.

Practical Recipe

1. Pick the output target deliberately: full video (rich but expensive), occupancy plus flow (planning-ready), or latent state (compact).
2. Condition generation on action and structure (map, boxes) so rollouts are controllable, not free-running dreams.
3. Evaluate fidelity over the planning horizon, not just one step; dynamics drift compounds.
4. Validate by closed-loop transfer: train or plan in the model, test on real or high-fidelity sim.
5. For augmentation, oversample rare events and verify downstream metrics improve on a real held-out set.