Section 48.3: Detection, lane and behavior prediction

To predict a driver is to admit you do not know what they will do, and then to be ready for every reasonable thing they might.

On trajectory forecasting

A car idling at a green light could go straight, turn left across your path, or sit there texting for another four seconds, and the only honest prediction is all three at once. Behavior prediction turns that discomfort into a measurable contract: ingest agent histories and a vectorized map, output \(K\) candidate future trajectories per agent with probabilities, and score them against the realized future. Because the future is uncertain, the standard metrics evaluate the best of \(K\) samples rather than a single guess, which rewards covering the true mode without penalizing diversity. A single-mode predictor on the Argoverse 2 benchmark produces a minFDE of roughly 3.5 m at a 6-second horizon; adding lane-graph conditioning and six diverse modes typically drops that to around 1.2 m (as of 2023), a factor-of-three reduction that in practice translates into earlier and safer planner responses at intersections.

Figure 48.3A captures the core intuition: a single observed history fans out into several plausible futures, and a good predictor keeps every one of them alive until the evidence resolves which will happen.

A predictor that collapses six futures into one has not simplified the problem; it has hidden the part that matters most when the road diverges.

Theory

The displacement metrics

Let an agent have ground-truth future positions \(y_1, \dots, y_T\) and \(K\) predicted trajectories, the \(k\)-th being \(\hat y_1^{(k)}, \dots, \hat y_T^{(k)}\). The minimum Average Displacement Error (ADE) over \(K\) samples averages the per-timestep L2 distance to the nearest predicted mode:

$$\text{minADE}_K = \frac{1}{T}\,\min_{k=1}^{K}\sum_{t=1}^{T}\big\lVert \hat y_t^{(k)} - y_t \big\rVert_2.$$

The minimum Final Displacement Error (FDE) keeps only the last timestep,

$$\text{minFDE}_K = \min_{k=1}^{K}\big\lVert \hat y_T^{(k)} - y_T \big\rVert_2.$$

Both take the best of \(K\) predictions: the model is credited if any one of its samples is close to the truth. A related metric, the miss rate, counts the fraction of agents for which even the best sample's final point exceeds a threshold (commonly 2 m).

Lane-graph-conditioned models

A vectorized lane graph matters in embodied AI because the vehicle body cannot teleport. Every predicted trajectory that exits a drivable lane forces the downstream route and local planner to discard it, which wastes compute and triggers last-instant replanning. At highway speeds a 50 ms planning delay corresponds to roughly 1.5 m of travel, so discarded off-road predictions directly lengthen reaction distance. Without map conditioning, the predictor treats the road surface as unconstrained free space, a physically false and hazardous assumption.

Vectorization converts each lane centerline into a polyline (a sequence of straight-line segments joining ordered points, here each segment stored as a short displacement vector) of short displacement vectors. A graph attention layer, a neural network layer that lets each node in a graph selectively weight information from its neighbors, connects adjacent and successor segments, so a query about one lane inherits turn-restriction and yield information from its neighbors. Encoding the agent history into the same vector space lets cross-attention, a mechanism where one set of features (here, the agent history) queries another set (the lane graph) to pull in relevant context, anchor each predicted mode to a lane the agent can actually reach. Figure 48.3B below traces this pipeline end to end.

VectorNet encodes both agent histories and map elements (lane centerlines, crosswalks, boundaries) as polylines of vectors. A graph network then reasons over these polylines so predictions respect road geometry. MTR (Motion Transformer) pairs learnable motion-mode queries with a transformer to produce diverse, map-consistent trajectories, and it led the Waymo motion-prediction benchmark as of 2023. Both architectures must fit inside the AV stack's 100 ms planning cycle. On Nvidia Drive AGX Orin (the compute platform used by multiple production AV programs), a six-mode MTR forward pass over 32 tracked agents typically completes in roughly 18 ms, which leaves the remainder for sensor fusion, planning, and control. This timing budget is why lane-graph-anchored multi-modal decoding matters beyond accuracy alone. Skipping the map encoder to predict free-space trajectories saves time on paper, but those trajectories routinely exit the drivable surface at intersections. The downstream planner then rejects or clips them, which erases any latency advantage. As an illustrative order of magnitude, without lane conditioning roughly 40 percent of predicted modes at a four-way intersection might leave the drivable surface and require discarding; with conditioning, that figure can drop below 2 percent, so the planner receives six usable hypotheses instead of three half-valid ones.

Worked Example

With the metric definitions and the decoding algorithm in hand, the fastest way to internalize why best-of-K rewards coverage is to compute it directly on a handful of trajectories.

The example computes \(\text{minADE}_6\) given six predicted trajectory samples and one ground-truth future.

import numpy as np

# Ground truth: T=5 future (x, y) positions of one agent.
gt = np.array([[1.0, 0.0], [2.0, 0.1], [3.0, 0.3], [4.0, 0.6], [5.0, 1.0]])

# Six predicted trajectories, shape (K=6, T=5, 2).
# Sample 0 hugs the truth; the rest are alternative modes (lane changes, stops).
rng = np.random.default_rng(0)
preds = np.stack([
    gt + rng.normal(0, 0.05, gt.shape),          # near-correct mode
    gt + np.array([0, 1.5]),                      # left lane change
    gt + np.array([0, -1.5]),                     # right lane change
    gt * np.array([0.6, 1.0]),                    # braking / slowing
    gt + rng.normal(0, 1.0, gt.shape),            # noisy
    gt + np.array([0.0, 3.0]),                    # far-off mode
])

def min_ade_k(preds, gt):
    """minADE_K: best-of-K mean per-timestep L2 displacement."""
    # Per-sample, per-timestep L2 distance: shape (K, T).
    disp = np.linalg.norm(preds - gt[None, :, :], axis=2)
    ade_per_sample = disp.mean(axis=1)            # average over T -> (K,)
    return float(ade_per_sample.min())            # best of K

def min_fde_k(preds, gt):
    """minFDE_K: best-of-K final-timestep L2 displacement."""
    final = np.linalg.norm(preds[:, -1, :] - gt[-1], axis=1)  # (K,)
    return float(final.min())

print("minADE_6:", round(min_ade_k(preds, gt), 3), "m")
print("minFDE_6:", round(min_fde_k(preds, gt), 3), "m")

Expected output: a small minADE_6 (roughly 0.04 m) and minFDE_6, because sample 0 closely tracks the truth. Remove sample 0 from the stack and both metrics jump, demonstrating that the score depends entirely on whether the correct mode was among the \(K\) candidates.

Practical Recipe

1. Define the horizon \(T\), the number of modes \(K\), and the miss threshold up front; they make numbers comparable.
2. Condition on the vectorized lane graph, not just agent history, so predictions stay on the road.
3. Report minADE_K, minFDE_K, miss rate, and a mode-coverage check together; one metric alone hides collapse.
4. Stress-test on rare maneuvers (cut-ins, sudden stops), not just the routine straight-ahead majority.
5. Save one artifact: per-agent predictions, probabilities, the realized future, and the four metrics.