Section 38.6: World models for visual control

"In hardware, the rejection decision is often more important than the nominal prediction."

A Latent State That Must Survive Contact

Problem First

A world model that looks great on benchmark rollouts can still fail the moment vision becomes ambiguous, latency spikes, or the robot makes contact. Visual control is the acid test because the learned state must handle high-dimensional sensing while preserving the low-dimensional geometry and timing that control depends on. A model that predicts the future beautifully but arrives too late to act on it is not a controller; it is a spectator.

Core Model

A deployed visual-control latent often combines multiple sensing streams: $$z_t = f_\theta\big(\mathrm{enc}_{\text{vision}}(o_t^{1:m}), \mathrm{enc}_{\text{prop}}(q_t, \dot q_t), a_{t-1}, h_{t-1}\big).$$ This matters because vision alone rarely resolves hidden contact state, while proprioception alone rarely resolves scene structure.

The control objective remains the same, but rollout error should now be read through a safety lens: $$a_t = \pi(z_t), \qquad \text{reject if } \Pr(\text{collision or instability} \mid z_t) > \tau.$$ In practice, the world model becomes one module in a larger stack that may include a low-level stabilizer, safety filter, or reflex policy. Figure 38.6B below traces this fuse-then-gate loop end to end.

Visual control therefore favors representations that are task-sufficient, multimodal, and timing-aware. A perfect decoder is optional. Stable contact prediction, fast inference, and recoverable failure handling are not.

Minimal Probe

The probe below fuses a visual embedding with proprioception and then checks whether a simple confidence gate would reject an unsafe rollout. The logic is intentionally small, because this is the boundary every hardware stack eventually needs to expose.

# Fuse vision and proprioception, then gate execution by confidence.
# The rejection decision is often more important than the nominal action.
import numpy as np

vision_latent = np.array([0.62, 0.18, 0.51])
proprio_latent = np.array([0.55, 0.24, 0.48])
fused = 0.7 * vision_latent + 0.3 * proprio_latent
uncertainty = np.abs(vision_latent - proprio_latent).mean()
execute = uncertainty < 0.08
print({"fused_state": np.round(fused, 3).tolist(), "uncertainty": round(float(uncertainty), 3), "execute": execute})

Expected behavior: Execution should proceed only when the sensing streams agree closely enough for the controller to trust the fused state. If uncertainty stays high during contact or occlusion, the stack needs a fallback mode rather than a stronger decoder.

Practical Recipe

1. Log vision-only, proprio-only, and fused-state diagnostics separately.
2. Define a rejection policy for latent uncertainty before the first hardware test.
3. Stress the model with lighting change, occlusion, calibration drift, and mild contact mismatch.
4. Measure wall-clock latency alongside task success; a stronger latent that arrives too late is still a failure.

Visual control is where world models stop being abstract. The useful representation must carry geometry, embodiment, and timing in one state. That often means a multimodal latent, a shorter imagination horizon than benchmark videos suggest, and an explicit contract for when to reject the model's advice.

Once that contract for rejecting advice is in place, the remaining design question is how much of the raw image the latent should bother to preserve at all. Teams must also decide whether to keep an image decoder. Some keep it because reconstructions reveal what the latent forgot. Others drop it and redirect capacity to reward, value, or contact heads. The right choice depends on two things: which failures the builder needs to diagnose, and how much inference budget the controller allows. TD-MPC2 (Hansen et al., 2023) takes the decoder-free path. It drops image reconstruction entirely and runs latent Model Predictive Control (MPC) at under 10 ms per step on a single GPU, achieving strong results across 104 continuous-control tasks. A decoder-based model at comparable scale typically requires 60-80 ms per step, which in practice makes it unusable for real-time servo loops without further optimization. DreamerV3 (Hafner et al., 2023) keeps the decoder and uses it to diagnose which scene parts the world model failed to encode. The reconstruction loss then doubles as a monitoring signal during deployment. This is the latency-fidelity tradeoff: decoder-free models are faster and more compact, but they lose the visual sanity-check that catches silent encoder failures.

This tradeoff matters physically because a robot servo loop tolerates only a fixed time budget per control step. At 30 Hz, each step has 33 ms; at 10 Hz, 100 ms. Pixel reconstruction that costs 60-80 ms leaves almost nothing for planning, communication, and actuation. On contact-rich tasks, arriving late with a beautiful prediction is worse than arriving on time with a coarser one: the joint has already moved.

The mechanism is direct. A decoder-based model back-propagates through pixel reconstruction, forcing the latent to retain spatial detail that control may never use. A decoder-free model back-propagates only through reward, value, or contact heads, so the latent compresses toward task-relevant geometry, and inference skips the decoder pass entirely. That cuts per-step compute by roughly 6-8x, and the freed budget buys planning depth: within the same 33 ms window, a decoder-based model's 5-step lookahead can become a decoder-free model's roughly 30-40-step lookahead (consistent with the 6-8x compute saving, and dependent on the specific model and hardware), turning a controller that sees one move ahead into one that sees several seconds of consequences.