Section 48.9: Closed-Loop Driving Evaluation And Safety Assurance

Open-loop scores measure how well you predict the world; closed-loop scores measure how the world changes once you act in it, and only the second kind keeps you safe.

On evaluating a driving stack

Open-Loop Metrics Are Necessary, Not Sufficient

Ask a driving stack the wrong question and it will answer confidently all the way into a stopped truck: score it on how well it predicts the world, and it never learns what happens when its own next move rewrites that world. Figure 48.9.1 maps the full chain this section follows to close that gap, from sensing and prediction through planning and control to the evidence artifacts that defend a deployment claim; keep it in view as the reference picture for everything below.

Modern driving stacks are usually evaluated in at least three modes. Open-loop evaluation checks perception or prediction against recorded data. Closed-loop simulation checks how the full stack behaves when its own decisions perturb the future. Safety assurance asks a different question again: which hazards remain, what mitigations exist, and what evidence shows that the mitigations actually work inside the stated operational design domain?

The first two modes are often confused. A forecasting model can improve average displacement error while still making the planner too confident in the wrong interaction pattern. One studied stack (circa 2023) dropped its open-loop displacement error by 18% across a merge benchmark, then posted a 31% higher collision rate in closed-loop because the planner read the sharper forecast as license to enter tighter gaps: this is the open-loop confidence trap, where better prediction directly causes worse driving. A smoother trajectory generator can reduce jerk (the rate of change of acceleration, felt by passengers as an abrupt jolt) while cutting safety margins. A benchmark score can rise even while the vehicle becomes easier to deadlock or easier to surprise in rare conditions.

Driving Assurance Mathematics

If acting on a decision changes the very distribution you are scored against, then a single scalar cannot carry that information, which is why the evaluation object has to be widened before it can be trusted.

Before going further, one term needs a definition because the rest of this section leans on it: a safety case (a structured, written argument, backed by evidence, that a system is acceptably safe for a stated use) is the document that ties benchmark numbers to a deployment decision. Closed-loop metrics, ODD bounds, and defeater analysis are not separate deliverables; they are the evidence sections inside that one safety case.

Closed-loop evaluation often reduces the stack to a score, but the useful object is a tuple of behavior and risk:

$$E_{\mathrm{drive}} = (c_{\mathrm{route}}, p_{\mathrm{infractions}}, m_{\mathrm{comfort}}, m_{\mathrm{interaction}}, d_{\mathrm{ODD}}).$$

Here \(c_{\mathrm{route}}\) is route completion, \(p_{\mathrm{infractions}}\) aggregates collisions and rule violations, \(m_{\mathrm{comfort}}\) tracks jerk and acceleration, \(m_{\mathrm{interaction}}\) tracks negotiation quality such as deadlock or unsafe gap choice, and \(d_{\mathrm{ODD}}\) records how close the scenario lies to the edge of the validated Operational Design Domain (ODD).

A useful benchmark formula is therefore not just "higher is better." One representative decomposition is:

$$J_{\mathrm{closed}} = c_{\mathrm{route}} \cdot p_{\mathrm{infractions}}, \qquad \text{subject to } d_{\mathrm{ODD}} \in \mathcal{D}_{\mathrm{validated}}.$$

This captures a core lesson from current leaderboard design: route progress matters, but only when infractions remain low and the scenario still sits inside the claimed ODD. Once the system leaves the validated domain, the benchmark score stops being a deployment argument and becomes a research observation.

The ODD As A Physical Contract

The ODD matters in embodied AI because physical systems cannot generalise away from the conditions they were validated on. A sensor suite calibrated for dry asphalt reflects light differently in standing water. A planner tuned on structured urban intersections has never encountered the timing patterns of a rural uncontrolled junction. A software service degrades gracefully, but a vehicle moving at speed suffers irreversible consequences when its perception or planning model operates outside its training distribution. The ODD is therefore a hard physical contract, not a marketing caveat. Consider why this boundary matters at scale. A rare lane-merge failure that appears once per 10,000 real-world miles would typically need on the order of 50,000 on-road episodes to characterise statistically. A tightly scoped closed-loop panel targeting that exact ODD slice can, in practice, surface the same failure in far fewer episodes (roughly 300 in illustrative estimates), because the simulator can hold most variables constant except the one being stressed; the exact ratio depends heavily on how faithfully the simulator reproduces the sensor and traffic conditions of the real failure.

Mechanically, ODD bounding tags every closed-loop scenario with the environmental and traffic parameters that define it, then checks whether those parameters fall within the validated envelope before the score becomes a deployment argument. At runtime, an ODD monitor ingests sensor diagnostics (LiDAR return rate, camera exposure confidence, map localisation error) and traffic context. It compares them against the declared envelope and either confirms that the system operates within bounds or triggers a fallback policy. A score computed outside the envelope goes into a separate log and counts as exploratory data rather than safety evidence.

Figure 48.9.2 lays out this evidence flow as a block diagram, tracing how open-loop tests, closed-loop driving runs, scenario coverage, and a safety case combine into a single deploy, limit, or deny decision.

Current Toolchain And What It Is For

Writing that defeater is only possible if you have tools that can stage the failing condition in the first place, so the assurance argument rests on a concrete toolchain, each part built to interrogate a different corner of the claim.

The practical tool stack for this section is: CARLA Leaderboard, nuPlan, Waymo Open Dataset, Waymo Sim Agents or Waymax, CommonRoad, Autoware, ASAM OpenSCENARIO, OpenDRIVE. Each tool answers a different assurance question. CARLA Leaderboard tests closed-loop route completion with explicit infraction metrics. nuPlan and CommonRoad structure planning evaluation. Waymo's public challenges and simulator tools expose interaction realism. Autoware makes stack interfaces concrete. OpenSCENARIO and OpenDRIVE keep the scenario and map assumptions portable.

Code And Evidence

Before reading the code below, ask yourself: if your stack scored 96% route completion with only one blocked-agent event, would you deploy it? And if not, what exactly would you need to see before you would?

A tiny assurance record puts this into practice. It does not compress the whole safety case into one object; it forces the stack to admit which metric improved, which infraction still happened, and whether the claim is promotable.

# Store one closed-loop assurance summary for a driving stack.
# Keep the route metric, infraction metric, and deployment decision together.
from dataclasses import dataclass, asdict

@dataclass
class AssuranceRecord:
    scenario_family: str
    route_completion_pct: float
    infraction_penalty: float
    blocked_events: int
    strongest_defeater: str
    decision: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = AssuranceRecord(
    scenario_family="unprotected_left_turn",
    route_completion_pct=96.0,
    infraction_penalty=0.91,
    blocked_events=1,
    strongest_defeater="late response to fast oncoming motorcycle in glare",
    decision="limit_to_daylight_until_glare_panel_passes",
)
print(record.as_row())

Expected output: the printed record should contain both performance and denial information, not just a success metric. If your assurance artifact cannot name the strongest remaining defeater and the resulting deployment limit, it is a benchmark summary rather than a safety argument.

Recipe For Builders

1. Write the ODD card before comparing models. A model that works in daylight urban traffic is not automatically a model for glare, heavy rain, rural roads, or emergency-vehicle interactions.
2. Use open-loop perception and prediction metrics as supporting evidence, then rerun the exact claim in closed-loop simulation.
3. Keep route completion, infractions, comfort, blocked-agent events, and intervention logs in one artifact.
4. Write the strongest defeater for every apparent success and convert it into a scenario or an explicit deployment limit.
5. Only promote the stack where the metrics and the written safety argument agree.

A score that cannot name the condition under which it breaks is not a safety argument; it is a number waiting for the road to finish the sentence.