Section 48.1: Driving as perception, prediction, planning, control

The hardest part of driving is not seeing the world or steering the wheel; it is the handoff in between, where one subsystem's confident output becomes another's silent assumption.

On the architecture of the modern driving stack

A self-driving car makes a life-or-death decision roughly every 70 to 100 milliseconds at a typical control rate, and no single stage of its software owns that decision: the label "autonomous driving" hides a closed loop that must ingest sensor streams, a map, and a route intent; estimate the state of the ego vehicle (the term the AV literature uses for "the car whose software is running this loop," to distinguish it from every other tracked vehicle) and of every relevant actor; commit to a trajectory; and convert that trajectory into steering, throttle, and brake commands at a fixed cycle rate. The loop is typically judged by route completion, collision rate, and safety margin, measured on the same scenarios for every candidate stack. As Figure 48.1A emphasizes, every arrow between those stages is itself an interface that can fail, which is where most of the engineering difficulty lives.

Real deployments run this loop at 10 to 20 Hz. A perception frame that arrives 150 ms late is not a slow frame; it is a wrong frame, because the planner will act on a world that has already moved. Treating the stack as a timed pipeline, not a static dataflow diagram, is the central discipline of the chapter. Figure 48.1.1 lays out these four stages and the loop they close through the world; keep it in view as the rest of the section walks each interface in turn.

Theory

Formally, driving is a partially observed Markov decision process run under hard real-time constraints. At cycle \(t\) the vehicle receives an observation \(o_t\) (camera frames, LiDAR sweeps, radar returns, GNSS (satellite positioning, e.g. GPS) or IMU (inertial measurement unit, an onboard accelerometer/gyroscope package)), estimates a state \(\hat s_t\) (ego pose and velocity plus a set of tracked actors with positions, velocities, and classes), commits to an action \(a_t\) (a trajectory or a direct control command), and transitions to \(o_{t+1}\). The decomposition into perception, prediction, planning, and control is an engineering factorization of this single MDP that makes each piece testable in isolation.

Partial observability matters in a physical vehicle because sensors have blind spots, occlusions, and range limits that computation alone cannot recover. A pedestrian hidden behind a parked truck is absent from \(o_t\) regardless of algorithm quality. The planner must therefore act under genuine uncertainty about the true world state \(s_t\), not just noisy measurement of a fully visible scene. Physical inertia makes this consequential. A 1 800 kg vehicle at 50 km/h needs roughly 20 m to stop, so decisions made on incomplete observations carry irreversible real-world consequences that differ fundamentally from replanning in simulation. To see the gap, note that the hidden pedestrian needs only three steps into the road before the vehicle is already inside its own braking distance. The planner must account for that pedestrian before any sensor ever sees them.

The POMDP (Partially Observable Markov Decision Process) machinery separates state estimation from decision-making. The perception stage maintains a belief over \(s_t\) from the sensor model \(p(o_t \mid s_t)\) and the motion model \(p(s_t \mid s_{t-1}, a_{t-1})\). It then collapses that belief into a point estimate \(\hat{s}_t\) with an associated uncertainty \(\Sigma_t\). The planner optimizes expected cost over the whole distribution rather than a single assumed state. This is why prediction produces a set of weighted trajectories rather than one deterministic forecast: the weights are the residual probability mass the planner must hedge against.

If each stage handles its piece so well, why do AV systems still surprise their engineers with failures? The factorization buys modularity at the cost of compounding error at every interface. A perception error (a false negative, an identity switch, localization drift) corrupts prediction's input; a prediction error (an unforecast lane change) corrupts planning's. So every interface needs its own metric, and the stack needs a closed-loop metric on top: passing each component benchmark does not guarantee a safe drive.

A model that works in simulation but fails on the road is not a model; it is a liability, because the road does not offer a reset button.

Worked Example

The fastest way to find where the timing contract and the four-stage handoffs break is to run the smallest loop that still contains all of them. The example below is a minimal closed-loop simulation that exercises all four stages on a single car-following scenario. It exposes the interface fields a real stack must log: the estimated state, the prediction, the planned trajectory, and the control command, plus the safety margin that turns the run into evidence.

import numpy as np

# Ego and a lead vehicle on a straight road (1D for clarity).
ego = {"s": 0.0, "v": 12.0}        # position (m), speed (m/s)
lead = {"s": 30.0, "v": 8.0}       # slower lead vehicle ahead
dt = 0.1                            # 10 Hz control loop
T_HORIZON = 2.0                     # prediction/planning horizon (s)
TIME_GAP = 1.5                     # desired time gap (s)

def perceive(ego, lead):
    """Return measured gap and lead speed (noisy in reality)."""
    return lead["s"] - ego["s"], lead["v"]

def predict(gap, lead_v):
    """Constant-velocity forecast of the lead over the horizon."""
    return gap + (lead_v - ego["v"]) * T_HORIZON  # predicted future gap

def plan(gap, lead_v):
    """Choose target speed to hold a safe time gap, in the style of the
    Intelligent Driver Model (IDM), a car-following rule that balances a
    desired time gap against the current closing speed."""
    desired_gap = max(5.0, TIME_GAP * ego["v"])
    accel = 1.0 * (lead_v - ego["v"]) + 0.5 * (gap - desired_gap)
    return np.clip(accel, -4.0, 2.0)   # m/s^2, bounded actuator

def control(accel):
    """Convert planned acceleration into a (bounded) command."""
    return float(np.clip(accel, -4.0, 2.0))

for step in range(30):
    gap, lead_v = perceive(ego, lead)
    future_gap = predict(gap, lead_v)
    accel = plan(gap, lead_v)
    cmd = control(accel)
    # Apply control, advance the world one cycle.
    ego["v"] = max(0.0, ego["v"] + cmd * dt)
    ego["s"] += ego["v"] * dt
    lead["s"] += lead["v"] * dt
    if step % 10 == 0:
        print(f"t={step*dt:4.1f}s gap={gap:5.1f}m future_gap={future_gap:5.1f}m "
              f"v={ego['v']:4.1f} cmd={cmd:+.2f}")

print("min safety margin (gap) held:", round(lead['s'] - ego['s'], 2), "m")

Expected output: the gap shrinks from 30 m toward the desired time-gap distance and then stabilizes; the command saturates at the actuator limit during the initial deceleration and the final safety margin stays positive. The run is only useful as evidence because it logs the per-stage fields and the safety margin, not just "did not crash."

Practical Recipe

The worked example gives you one loop that runs; the steps below turn that single run into a repeatable discipline for building and stress-testing any stack.

1. Write the stack contract: the observation, the state estimate, the action interface, the cycle rate, and the safety metric.
2. Build the smallest closed loop that can fail interpretably (the car-following example above).
3. Add latency to one stage and measure how the safety margin degrades; this localizes timing brittleness.
4. Replace one stage with the library version (for example a learned detector) and re-run on identical scenarios.
5. Save one artifact: config, seeds, per-stage logs, summary metrics, and two representative traces (one nominal, one near-miss).