Section 41.2: Diffuser and Decision Diffuser

Do not predict the next action; predict the next chunk of actions, then denoise the whole chunk until it is consistent with what the camera just saw.

An Action-Diffusion Policy

Diffuser and Decision Diffuser solve related but different planning problems, as Figure 41.2A contrasts: Diffuser samples trajectories that satisfy start-state and goal constraints, while Decision Diffuser samples trajectories that are also conditioned on desired return or value targets. Both belong to a broader family of action-diffusion methods whose practical robotics workhorse is Diffusion Policy: instead of denoising a full offline trajectory, it denoises a short chunk of future actions conditioned on the latest visual observation, then executes part of that chunk before replanning.

For robotics, the difference between these methods matters whenever the task objective is richer than "reach the goal." Return conditioning can favor high-reward but risky plans unless it is paired with explicit feasibility or safety filtering, and observation-conditioned action chunking trades a longer planning horizon for tight, reactive control at execution time.

A trajectory that looks coherent in noise space is not yet a plan; it becomes one only after the full denoising chain agrees on every joint angle at every time step. Figure 41.2B diagrams this reverse chain: a fixed condition steers the noise tensor through intermediate partial trajectories until it settles into a coherent plan.

Theory

Action-diffusion methods are trained with the denoising score-matching loss. Given a clean action sequence \(\tau_0\) conditioned on observation \(o\), sample a timestep \(t\) and Gaussian noise \(\epsilon\), form the noised sample using the forward closed form (a single algebraic formula that jumps directly from the clean trajectory to its noisy version at step \(t\), avoiding \(t\) sequential noising steps), and ask the network to predict the noise it sees:

$$ \mathcal{L} = \mathbb{E}_{t,\tau_0,\epsilon}\left[\left\lVert \epsilon - \epsilon_\theta\!\left(\sqrt{\bar\alpha_t}\,\tau_0 + \sqrt{1-\bar\alpha_t}\,\epsilon,\ t,\ o\right)\right\rVert^2\right]. $$

This is a regression on noise, not on actions, which is what lets the model represent a multimodal action distribution: many distinct \(\tau_0\) values are consistent with the same observation, and the noise objective does not collapse them to an average. Diffuser uses the same loss over full trajectories \(\tau=(s_0,a_0,\dots,s_H,a_H)\) conditioned on state or goal; Decision Diffuser adds a return target \(\hat R\) to the conditioning \(c\), turning "sample a plausible future" into "sample a plausible future likely to achieve this return."

How the return target actually steers sampling. Decision Diffuser trains the denoiser twice, once with the return conditioning present and once with it dropped (replaced by a null token), a scheme called classifier-free guidance. At sampling time, the model computes both the conditioned noise prediction \(\epsilon_\theta(\tau_t, t, c)\) and the unconditioned prediction \(\epsilon_\theta(\tau_t, t, \varnothing)\), then extrapolates away from the unconditioned direction: \(\hat\epsilon = \epsilon_\theta(\tau_t, t, \varnothing) + w\,\bigl(\epsilon_\theta(\tau_t, t, c) - \epsilon_\theta(\tau_t, t, \varnothing)\bigr)\), where guidance weight \(w>1\) pushes the sample further toward trajectories the conditioned model favors and \(w=1\) recovers plain conditional sampling. This is the concrete mechanism behind the "dial in a desired return" claim in the Big Picture callout above: raising \(w\) trades sample diversity for stronger adherence to the requested return, and it is why the Practical Recipe below advises clipping the requested return before sampling, since a high \(w\) combined with an out-of-distribution \(\hat R\) is what produces the extrapolation failures described in the Common Pitfall callout.

Once the return dial has biased the denoiser toward the futures you want, the remaining question is how quickly you can turn that trained denoiser into an actual sample. The two sampling schemes below, DDPM and DDIM, are the prerequisite for everything that follows: action chunking, the inference algorithm, and the worked example all assume you have already chosen one of them. Sampling: DDPM vs DDIM. You can sample the trained denoiser two ways. Denoising Diffusion Probabilistic Models (DDPM) run the full stochastic reverse chain, typically hundreds to a thousand steps, and inject fresh noise at each step. Denoising Diffusion Implicit Models (DDIM) reinterpret the same model as a deterministic (non-Markovian) trajectory from noise to data, so you can skip steps and sample in roughly 10 to 50 iterations with little quality loss. The gap is stark. A 1000-step DDPM pass takes roughly 500 ms on a typical GPU. A 10-step DDIM pass on the same model takes about 5 ms, enough to close a 200 Hz control loop. DDPM gives more sample diversity; DDIM gives the speed a control loop needs. Consider the stakes at 0.3 m/s wrist velocity. A Franka arm travels 150 mm during a single 500 ms DDPM pass, roughly the diameter of the object it is trying to grasp, so the plan is already stale the moment it arrives. The 5 ms DDIM pass lets the arm travel less than 2 mm between plans, keeping the trajectory and the real world synchronized.

Action chunking. Rather than denoising one action, Diffusion Policy predicts a horizon of \(H\) future actions in a single denoising pass, executes the first \(k < H\) of them open-loop, then re-observes and replans. Predicting a chunk gives temporal consistency across the action horizon (no jitter between consecutive single-step predictions); executing only \(k\) and replanning every \(k\) keeps the policy reactive to disturbances. The choice of \(H\) and \(k\) is the main knob trading smoothness against responsiveness, mirroring the horizon-versus-reactivity trade-off in receding-horizon MPC, where MPC (model predictive control) is the pattern of re-solving for a full control sequence at every step but executing only its first portion.

Without chunking, a single-step policy queries the denoiser at every control tick. On a 50 Hz Franka controller, consecutive single-step predictions share no explicit coupling. Joint velocities can therefore reverse between ticks, and those reversals produce torque spikes that trip safety limits or stress actuators. Chunking denoises the entire planned arc jointly, so adjacent time steps correlate by construction rather than by chance.

The joint noise tensor produces temporal consistency. The denoiser receives the full \((H \times A)\) noise array and reduces all \(H\) steps at once in each reverse iteration. Every denoising step uses the same observation embedding and processes all time steps together. That shared forward pass lets the network enforce smooth velocity profiles. Steps at index 3 and 4 share the same noise tensor, so they cannot contradict each other.

Worked Example

The tiny two-step trace above moved single numbers by hand; the probe below scales that same reverse-chain logic up to a full action chunk in code. It shows how to structure a Diffusion Policy inference loop. It denoises an action chunk of shape \((H, A)\) from Gaussian noise over \(T\) DDIM steps, conditioning every step on a fixed observation embedding, then executes the first \(k\) actions. The denoiser here is a stand-in; in a real system it is a trained U-Net (a convolutional network with a contracting-then-expanding shape and skip connections between matching resolutions) or transformer. The control logic around it, start from noise, iterate the reverse step, condition on the observation, slice the first \(k\), is exactly what production code does.

# Diffusion Policy inference: denoise an action chunk conditioned on an observation.
# Loop structure is the point; eps_theta stands in for a trained noise predictor.
import numpy as np

H, A = 8, 2                              # predict H future actions of dim A
T = 10                                   # DDIM denoising steps
k = 2                                    # execute the first k, then replan
abar = np.linspace(0.95, 0.02, T)[::-1]  # signal retention, clean -> noisy
obs = np.array([0.6, -0.2])             # observation embedding (stand-in)
rng = np.random.default_rng(1)

def eps_theta(a_t, t, o):
    # Trained model would go here. Stand-in: actions track the observation.
    target = np.tile(o, (H, 1))
    return (a_t - np.sqrt(abar[t]) * target) / np.sqrt(1.0 - abar[t] + 1e-8)

a_t = rng.normal(size=(H, A))           # start from pure Gaussian noise
for t in reversed(range(T)):            # reverse chain: t = T-1 ... 0
    eps = eps_theta(a_t, t, obs)        # predict the noise, conditioned on obs
    a0_hat = (a_t - np.sqrt(1.0 - abar[t]) * eps) / np.sqrt(abar[t])
    if t > 0:                           # DDIM deterministic step toward a_{t-1}
        a_t = np.sqrt(abar[t-1]) * a0_hat + np.sqrt(1.0 - abar[t-1]) * eps
    else:
        a_t = a0_hat                    # final step lands in action space

action_chunk = a_t
print("predicted chunk (first 3 of H):", action_chunk[:3].round(3).tolist())
print("execute first k =", k, "actions:", action_chunk[:k].round(3).tolist())

The chunk converges to the observation-consistent target only because the stand-in denoiser drives it there; a trained \(\epsilon_\theta\) would produce a multimodal, observation-appropriate sequence. Three structural points survive that swap: the observation is embedded once and reused across all \(T\) steps, DDIM keeps \(T\) small enough for control rates, and the loop executes only \(k\) of the \(H\) predicted actions before re-observing.

Practical Recipe

1. Pin the observation, action, and control rate first. For a Franka Panda arm, this means specifying whether the observation is a 640x480 RGB frame at 30 Hz, proprioceptive joint angles at 1 kHz, or a fused wrist-force plus depth image. Diffusion Policy on a Franka running pick-and-place from the Chi et al. CoRL 2023 setup uses a 10-step DDIM schedule to stay under the 20 ms budget imposed by the 50 Hz Cartesian impedance controller. If your planner exceeds that budget, the controller reads stale actions and contact forces build up unpredictably at the fingertip.
2. Start with a behavior-cloning baseline on the same demonstration data so you can separate gains that come from the diffusion architecture from gains that come from dataset quality or observation preprocessing. On the LeRobot benchmark, a CNN-BC baseline on 50 demonstrations of a push-T task gives a reliable lower bound before layering action chunking.
3. Choose the action horizon \(H\) and execute count \(k\) to match your physical loop. Chi et al. report that \(H=16\), \(k=8\) works well for tabletop push tasks on a UR5 with a wrist camera, but longer horizons accumulate drift in the presence of rigid contact because the denoised chunk cannot anticipate the reaction force mid-sequence. Shorter \(k\) (replanning more often) helps when objects shift unpredictably; longer \(k\) reduces jitter from frequent re-embedding of the visual observation.
4. Log the per-step denoising time, the action norm, and any constraint violations in the same artifact. A sudden spike in action norm often signals that the DDIM schedule was not re-indexed after changing the inference step count (see the Tip callout above), not a dataset problem. Keep the raw trajectory tensor next to the metric so failures can be replayed offline in MuJoCo or Isaac Sim without hardware access.
5. Run at least one contact-perturbation test before declaring success. Push the target object 2 cm off its expected position mid-episode and verify that replanning within one receding-horizon window (\(k\) steps) recovers without a joint-limit violation. On a 7-DoF arm this is the minimum sanity check before moving to sim-to-real transfer, because diffusion planners trained purely on scripted demonstrations often produce wrist-flip motions when the object pose drifts outside the training support.

Diffuser is the cleaner choice when you want a multimodal proposal distribution over trajectories and you already have an external scorer or feasibility filter. Decision Diffuser is stronger when the return target is meaningful and stable enough to guide sampling, but it becomes fragile when reward shaping or offline support is poor.

For robotics, the best practice is to treat both as candidate generators, not final arbiters. Let them propose futures, then let explicit collision checks, dynamics checks, and latency budgets decide what can really be executed.

Keep one inspectable probe for the model assumption, then use maintained libraries without changing the artifact schema used for baseline comparison.

1. Write the observation, action, state estimate, success metric, and rejection criterion.
2. Run a deterministic smoke test on one seed and save the complete configuration.
3. Add one perturbation tied to the section topic: delay, noise, horizon length, contact change, distractor object, or generated-scene shift.
4. Compare only methods evaluated by the same script, split, seed panel, and metric definition.
5. Record a postmortem that assigns failures to perception, representation, dynamics, planning, control, data coverage, timing, or evaluation.

When Diffuser and Decision Diffuser fails, do not collapse the result into a single method verdict. Assign the failure to the interface that broke, rerun one controlled perturbation, and keep the trace next to the metric. That habit turns a disappointing rollout into a reusable diagnostic asset.