Section 41.3: Generative trajectory planning and scoring

Stop learning to reverse a thousand noising steps; learn the straight-line velocity that carries noise to an action in one smooth flow.

A Flow-Matching Action Expert

What if a robot could turn a burst of random noise into a precise reaching motion in a single smooth glide, no hundred-step unwinding required? Flow matching does exactly that: instead of learning to reverse a noising process step by step as diffusion does, it learns a velocity field that transports a noise sample to a data sample along a smooth path, sampled by integrating an ordinary differential equation (ODE). For action generation this matters because the ODE can be integrated in a handful of steps, and the training target is a clean regression on velocity rather than on noise across a long schedule. Figure 41.3A shows the overall arrangement this section builds toward: a generator samples a diverse batch of candidate trajectories from the scene observation, a discriminative scorer ranks them by expected task success, and the top-ranked feasible trajectory is executed. The Theory section below defines that discriminative scorer formally, as the source of the reward term \(R\) in the composite scoring rule, before the algorithm puts it to use.

Flow matching for action generation connects to the scoring problem that any generative planner must still solve. A flow-matching expert proposes actions cheaply; receding-horizon execution then scores and filters those proposals at every observation refresh rather than once at episode start.

Theory

Flow matching defines a time-indexed path \(x_t\) for \(t\in[0,1]\) that starts at a noise sample \(x_0\sim\mathcal{N}(0,I)\) and ends at a data sample \(x_1\) (an action or action chunk). A learned velocity field \(v(x_t, t, c)\) generates the sample by integrating the ODE

$$ dx_t = v(x_t, t, c)\,dt, $$

from \(t=0\) (noise) to \(t=1\) (action), conditioned on context \(c\). Conditional Flow Matching chooses the simplest possible path: a straight line between a paired noise sample and data sample, \(x_t = (1-t)\,x_0 + t\,x_1\). Differentiating gives a constant target velocity \(x_1 - x_0\), so the training objective is the regression

$$ \mathcal{L}_{\text{CFM}} = \mathbb{E}_{t, x_0, x_1}\left[\left\lVert v_\theta(x_t, t, c) - (x_1 - x_0)\right\rVert^2\right]. $$

This is markedly simpler than DDPM training: no noise schedule to tune, no variance terms, and straight-line paths that integrate accurately in few Euler steps (the simplest way to numerically follow an ODE: repeatedly nudge the current point by velocity times a small step size, \(x \leftarrow x + v\,\Delta t\), rather than solving the equation exactly). In practice, a Denoising Diffusion Probabilistic Model (DDPM) chain typically requires around 100 denoising steps to produce one action; a flow-matching expert typically reaches comparable quality in 4 to 10 steps, cutting inference latency by roughly 10 to 25x in the reported comparisons and making real-time replanning at 20 Hz feasible where it was not before (as of 2024). These figures come from published few-step flow-matching versus DDPM comparisons on robot-action benchmarks and should be read as typical ranges rather than a universal guarantee, since exact speedups depend on the action dimensionality and the specific hardware.

A single reward term is insufficient for embodied AI. A robot lives in a physical world where task success, collision avoidance, actuator limits, and execution time are separate concerns that conflict. A trajectory that maximizes task reward may still break a joint limit, clip an obstacle, or demand torques the hardware cannot produce. On real hardware each failure mode carries a different cost. A collision can damage the robot or environment. A dynamics mismatch makes the executed motion diverge from the planned one and compounds error over later steps. Ablations on tabletop manipulation illustrate the consequence, though the exact episode counts are specific to that benchmark and setup rather than a general law. In one such ablation, a generator trained against a single task-reward signal needed roughly 40,000 demonstration episodes before collision-free success rates held above 80 percent, while adding a separate collision-penalty term to the scorer cut that threshold to around 600 episodes, because the generator no longer had to infer physical safety from sparse reward alone. Multi-term scoring makes these trade-offs explicit and separately tunable instead of hoping one learned reward captures every physical constraint.

Scoring operates as a two-stage filter. First, the planner evaluates \(R\), \(C\), \(D\), and \(L\) independently for each of the \(K\) candidates, weights each term by its \(\lambda\), and sums them into a scalar \(S(\tau^{(k)})\). A signed-distance field (a grid or function that stores, at every point in space, the distance to the nearest obstacle surface, with the sign marking inside versus outside) or lightweight collision checker supplies \(C\). The dynamics term \(D\) compares predicted joint velocities against the robot's measured velocity limits. The effort term \(L\) penalizes total path length or time. The reward term \(R\) is the one term that cannot be measured directly on an un-executed candidate, since the robot has not yet taken the action; in practice \(R\) comes from a learned discriminative scorer network trained to predict eventual task success from the observation and the proposed trajectory, the same discriminative scorer shown ranking candidates in Figure 41.3A, so \(R\) is itself a model output rather than a ground-truth measurement. Second, a hard feasibility gate discards any candidate whose \(C\) exceeds threshold \(\epsilon\), regardless of its overall score. A high reward never overrides a safety constraint. The planner then ranks the surviving candidates by \(S\) and executes the top one.

Worked Example

The probe below shows a flow-matching forward pass and a few-step ODE integration. Interpolate linearly between a noise sample \(x_0\) and a target action \(x_1\), read off the constant target velocity \(x_1 - x_0\), then integrate the ODE from noise to action with a handful of Euler steps. In a trained system the learned field \(v_\theta(x_t, t, c)\) replaces the constant velocity, but the path geometry and the integration loop are identical.

# Conditional flow matching: straight-line path from noise to action.
# x_t = (1 - t) x0 + t x1 ;  target velocity v = x1 - x0  (constant along the line).
import numpy as np

rng = np.random.default_rng(2)
x1 = np.array([1.0, 0.0])               # target action (data sample)
x0 = rng.normal(size=2)                 # noise sample

print("forward path and target velocity:")
for t in np.linspace(0.0, 1.0, 6):
    x_t = (1.0 - t) * x0 + t * x1       # point on the path at time t
    v = x1 - x0                         # CFM regression target
    print(f"  t={t:.1f}  x_t={x_t.round(3).tolist()}  v={v.round(3).tolist()}")

# Inference: integrate dx = v dt from t=0 (noise) to t=1 (action) with Euler steps.
x = x0.copy()
dt = 0.2
for _ in range(5):
    v_theta = x1 - x0                   # a trained v_theta(x, t, c) would go here
    x = x + v_theta * dt
print("integrated endpoint:", x.round(3).tolist(), " target:", x1.tolist())

The velocity is constant along the straight path, which is the whole appeal: the network only has to regress a stable target, and five Euler steps reach the action exactly. Replace the constant velocity with \(v_\theta(x_t, t, c)\) and the same loop becomes a real action sampler. This is the same mechanism behind flow matching for actions in imitation learning. The proposed action still enters the scoring stack, so raw proximity to a target is never the final word: an embodied planner only commits once feasibility and effort penalties agree.

Practical Recipe

1. Fix the observation representation before training: for a Franka Panda wrist-camera setup, concatenate the 84x84 RGB crop, the 7-DOF joint positions, and the 6-DOF end-effector pose into a single context vector \(c\). Feeding raw pixel tensors without cropping inflates the flow-matching regression loss and slows convergence by 2 to 3x in practice.
2. Set \(K\) (candidate count) relative to your control-loop budget. At 20 Hz on a Jetson AGX Orin, generating \(K=64\) candidates with 8 Euler steps fits within the 50 ms window; at 10 Hz (common for mobile-manipulation bases) you can afford \(K=128\). Profile on the actual inference hardware before committing to a candidate budget in training configurations.
3. Normalize the composite scorer before tuning \(\lambda\) weights. Collision penalty \(C\) derived from a signed-distance field is typically in meters (range 0 to 0.5 for a cluttered table), while task reward \(R\) from a binary success signal sits in \(\{0, 1\}\). Divide each term by its standard deviation over 200 rollouts from the Open X-Embodiment Bridge dataset split; only then tune the \(\lambda\) ratios.
4. Run the full proposal-score-filter cycle on one recorded episode from the RT-X dataset before any live robot trial. Log the candidate set, per-candidate scores, and the feasibility filter outcome as a single JSON artifact per episode. A planner that rejects all candidates on the logged episode will do the same on hardware.
5. Before trusting real-robot results, inject one contact perturbation: shift the target object 3 cm laterally from its nominal pose. A flow-matching expert trained on Diffusion Policy-style demonstrations should still produce a feasible candidate in at least 80 percent of trials; drop below that and the scorer is likely rejecting good plans due to a miscalibrated collision penalty threshold \(\epsilon\).

The right mental model is propose, score, then commit. Let the generator provide diversity, then use fast learned scorers and hard geometric checks to remove plans that are dynamically or spatially impossible. This usually beats asking the diffusion model alone to internalize every safety rule. Figure 41.3B lays out this propose-score-filter-commit pipeline as a diagram, including the receding-horizon feedback loop that re-plans on every new observation.

This split is also the cleanest architecture. Because the sampled plans, the scorer output, and the feasibility filter are inspectable separately, failure attribution is far easier than with a single opaque planner.

Each of those three inspectable stages maps onto a concrete piece of tooling, so the next step is to see which libraries own which part of the propose-score-filter pipeline.

Keep one inspectable probe for the model assumption, then use maintained libraries without changing the artifact schema used for baseline comparison.

1. Write the observation, action, state estimate, success metric, and rejection criterion.
2. Run a deterministic smoke test on one seed and save the complete configuration.
3. Add one perturbation tied to the section topic: delay, noise, horizon length, contact change, distractor object, or generated-scene shift.
4. Compare only methods evaluated by the same script, split, seed panel, and metric definition.
5. Record a postmortem that assigns failures to perception, representation, dynamics, planning, control, data coverage, timing, or evaluation.

That disciplined logging pays off most at the moment something breaks, because it lets you locate the fault instead of guessing. When Generative trajectory planning and scoring fails, do not collapse the result into a single method verdict. Assign the failure to the interface that broke, rerun one controlled perturbation, and keep the trace next to the metric. That habit turns a disappointing rollout into a reusable diagnostic asset.