Section 41.4: Generating scenes and synthetic experience

Generate the whole future as one object, then bend it toward high reward with a guidance gradient; the plan and the optimizer become the same forward pass.

A Trajectory-Level Planner

Ask a diffusion model to imagine ten thousand kitchens it has never touched, keep only the ones a real arm could survive, and you have replaced six months of robot teleoperation with an overnight GPU run. That is the bargain this section unpacks, and it starts from one observation: a generative planner is really just a sampler. Diffuser (Janner et al., 2022) generates a full trajectory at once by diffusion, then steers that generation toward high reward with classifier guidance (a separately trained reward or value model whose gradient nudges each denoising step toward higher-reward trajectories), so planning and optimizing collapse into the same denoising forward pass. Decision Diffuser instead conditions on a desired return-to-go (RTG, the sum of future rewards from the current timestep) for offline reinforcement learning. It samples trajectories that the dataset suggests will achieve that return. The same generative machinery also produces candidate futures and synthetic scenes that planners and learners can consume, as Figure 41.4A illustrates: a scene diffusion model conditioned on a task and an initial pose synthesizes environments and contact configurations whose resulting episodes augment the training set.

The catch is shared across all of these uses: generated trajectories and scenes help only when they stay near the real control regime, and the guidance that makes them high-reward can also pull them off the data manifold (the thin region of state-action space that real, physically valid trajectories actually occupy). The right framing is generation as a proposal engine whose proposals must survive a feasibility check, whether the proposal is a planned trajectory or a synthetic training scene.

Theory

Diffuser denoises an entire trajectory \(\tau\) and biases sampling toward high reward using classifier guidance or classifier-free guidance (a variant that trains one network for both conditional and unconditional predictions, eliminating the need for a separate classifier). At each reverse step the unconditional noise prediction is combined with a conditional one to form a guided estimate

$$ \tilde\epsilon_\theta = (1+w)\,\epsilon_\theta(\tau_t, t, c) - w\,\epsilon_\theta(\tau_t, t), $$

where \(c\) is the conditioning (a goal or a high-return signal) and \(w\) is the guidance weight. With \(w=0\) the sample is unconditional; larger \(w\) pushes harder toward the conditioned objective. The trade-off is direct: strong guidance shapes more goal-directed plans but can drag samples off the data manifold into dynamically infeasible trajectories, which is why a feasibility filter still sits downstream. Decision Diffuser uses this same conditioning mechanism with the return-to-go as \(c\), turning offline reinforcement learning into conditional trajectory generation.

Before leaning on that dial in a real loop, it is worth counting what each guided sample costs. A practical cost note frames the rest of the chapter: generative planning pays 20 to 50 denoising steps at inference for every plan, against a single forward pass for a deterministic policy. That latency is the price of multimodality and guidance, and it is exactly what the speedup work in Section 41.5 attacks.

The same guidance dial that steers a single plan becomes a data-generation lever once trajectories are banked instead of consumed one at a time. When generated trajectories are reused as training data, the loss mixes real trajectories \(\mathcal{D}_r\) with generated ones \(\mathcal{D}_g\): \(\mathcal{L}(\theta)=\mathbb{E}_{\mathcal{D}_r}[\ell_\theta] + \alpha\,\mathbb{E}_{\mathcal{D}_g}[\ell_\theta]\). The hard part is not forming that mixture. The hard part is ensuring \(\mathcal{D}_g\) covers task-relevant but underrepresented states instead of injecting trajectories that violate real embodiment dynamics. Too much synthetic data pushes the policy toward configurations that never appear on actual joints, surfaces, or sensors. That shift causes silent degradation at deployment. To tune \(\alpha\), hold out a hardware evaluation set and train policies at several values of \(\alpha\) on the same real-plus-synthetic mix. Select the value that minimizes the transfer gap. Monitor the verifier rejection rate at the same time: a low transfer gap achieved by discarding most synthetic proposals provides no coverage benefit.

Reading the mechanism above answers how guidance shapes a single plan, but this section's title promises scene and dataset-level generation, not just trajectory denoising. The rest of the section builds toward that promise directly: the worked example below isolates the guidance-weight trade-off on a single endpoint, the Practical Recipe then scales the same guided-generation loop to full scenes conditioned on real-data coverage gaps, and the algorithm box further above specifies exactly how accepted scene proposals get folded into a training mixture with the real dataset. Each step is a prerequisite for the next: understand the guidance dial on one trajectory before trusting it to fabricate whole scenes, and understand feasibility filtering before trusting the mixture weight \(\alpha\).

Worked Example

The probe below isolates the guidance-weight trade-off Diffuser depends on. From one noisy start it runs a guided denoising loop at three values of \(w\), blending a conditional denoiser (pulls the endpoint to the goal) with an unconditional one (pulls toward an on-manifold prior), and reports the endpoint, its distance to the goal, and whether it stays inside a feasible reach radius.

# Guided trajectory denoising (Diffuser-style).
# eps_tilde = (1 + w) eps_cond - w eps_uncond ; w trades goal-pull vs on-manifold pull.
import numpy as np

H = 5
goal = np.array([1.0, 0.0])
feasible_radius = 1.15           # endpoints beyond this are dynamically infeasible

def eps_uncond(tau):             # stand-in: pull toward a smooth on-manifold prior
    prior = np.stack([np.linspace(0.0, 0.6, H), np.zeros(H)], axis=1)
    return tau - prior

def eps_cond(tau, c):            # stand-in: pull the trajectory toward the goal c
    target = np.stack([np.linspace(0.0, c[0], H), np.linspace(0.0, c[1], H)], axis=1)
    return tau - target

for w in [0.0, 1.0, 4.0]:
    rng = np.random.default_rng(4)        # same start for every w
    tau = rng.normal(size=(H, 2)) * 0.3
    for _ in range(8):                     # guided reverse steps
        eps_tilde = (1 + w) * eps_cond(tau, goal) - w * eps_uncond(tau)
        tau = tau - 0.2 * eps_tilde
    endpoint = tau[-1]
    dist = float(np.linalg.norm(endpoint - goal))
    feasible = float(np.linalg.norm(endpoint)) <= feasible_radius
    print(f"w={w:>3}: endpoint={endpoint.round(3).tolist()} "
          f"dist_to_goal={dist:.3f} feasible={feasible}")

The three rows are the whole story of guidance. At \(w=0\) the plan stays comfortably feasible but stops short of the goal; at \(w=1\) it reaches the goal and is still feasible; at \(w=4\) it overshoots far past the feasible reach radius. This is why Diffuser pairs guidance with a feasibility check: turning \(w\) up makes plans more goal-directed right up to the point where they leave the manifold of trajectories the robot can actually execute. Decision Diffuser swaps the goal condition for a return-to-go target, but the same dial and the same risk apply.

Practical Recipe

1. Anchor the generator to a physical platform: specify the robot (Franka Panda, UR5, Spot), the sensor suite (RGB-D at 30 Hz, wrist-mounted F/T sensor, a force-torque sensor that measures contact forces and moments at the wrist, at 500 Hz), and the simulator (Isaac Sim, MuJoCo, PyBullet) before choosing a diffusion architecture.
2. Identify the real-data coverage gap by computing state-space density over your existing dataset. For tabletop manipulation, this typically means scenes in Open X-Embodiment or RT-X, large cross-institution datasets of real robot teleoperation episodes spanning dozens of robot embodiments, where specific object poses or clutter configurations appear fewer than 50 times.
3. Set the guidance weight w by sweeping from 0.5 to 4.0 on your longest intended horizon first. For a Franka Panda with H=20 steps at 10 Hz, values above 2.5 routinely push endpoints outside the 855 mm reach envelope and should be rejected by the geometric verifier before entering training.
4. Apply a three-stage verifier: (a) geometric feasibility (no interpenetration, contact normals inward, within the reachability map, the precomputed set of end-effector poses the arm can physically reach given its joint limits), (b) dynamic plausibility (joint velocity limits, torque bounds), and (c) a sim round-trip where MuJoCo executes the trajectory and measures drift from the generated keyframes (how far the simulated poses diverge from the diffusion model's proposed waypoints when real physics is applied).
5. Save one artifact per augmentation run: real-sample count, synthetic-sample count, guidance weight, verifier rejection rate, mixture weight alpha, and transfer gap (policy success rate on hardware minus success rate on sim held-out split). Without the rejection rate, a low transfer gap may simply mean the verifier discarded all informative proposals.

Synthetic data helps when three conditions hold together: real coverage is sparse in the target region, the generator can be conditioned on that region specifically, and a verifier (simulator check, geometric filter, or learned realism classifier) can reject off-manifold proposals before they enter training. The payoff when all three hold can be striking, though the exact ratio is task- and platform-dependent rather than a fixed constant. In practice, teams targeting rare contact failures have reported cutting the required real demonstrations from roughly 2,000 to under 200 by filling coverage gaps with verified synthetic scenes, a figure that should be read as an illustrative order of magnitude rather than a guaranteed rate; it typically holds only when the coverage gap is narrow and the verifier rejection rate is tracked alongside it. When that order of magnitude does hold, it represents something like a 10x reduction in real-data cost for the same downstream policy success rate. Put concretely, that gap is the difference between six months of daily robot teleoperation and a single two-week data collection sprint, with the diffusion generator doing the rest overnight on a GPU cluster. When any one of these is missing, the mixture tends to hurt. A generator conditioned on a broad return signal fills in high-return but physically impossible transitions. A verifier that only checks visual realism passes scenes with violated contact constraints. An unconditioned generator just oversamples the already-dense part of the distribution. The practical signal to watch is transfer gap: if a policy trained on real-plus-synthetic performs worse on hardware than one trained on real alone, the synthetic component is likely adding distributional shift rather than coverage. The most reliable pattern is to use generation as a data proposal engine and keep a conservative verifier downstream. Scene generators propose clutter, lighting, camera poses, or long-horizon futures; simulators, geometric filters, and held-out hardware tests decide whether those proposals are worth learning from.

For teaching and production alike, insist on one artifact that records real-sample count, synthetic-sample count, weighting, realism checks, and transfer outcome. Without that bookkeeping, synthetic data stories are nearly impossible to audit.

Keep one inspectable probe for the model assumption, then use maintained libraries without changing the artifact schema used for baseline comparison.

1. Write the observation, action, state estimate, success metric, and rejection criterion.
2. Run a deterministic smoke test on one seed and save the complete configuration.
3. Add one perturbation tied to the section topic: delay, noise, horizon length, contact change, distractor object, or generated-scene shift.
4. Compare only methods evaluated by the same script, split, seed panel, and metric definition.
5. Record a postmortem that assigns failures to perception, representation, dynamics, planning, control, data coverage, timing, or evaluation.

When Generating scenes and synthetic experience fails, do not collapse the result into a single method verdict. Assign the failure to the interface that broke, rerun one controlled perturbation, and keep the trace next to the metric. That habit turns a disappointing rollout into a reusable diagnostic asset.