Section 54.6: Deployment approval and safety cases

Approval is evidence traceability, not confidence performed in prose.

A Safety-Critical Controls Researcher

Why This Matters

Picture the moment a release board signs off on a robot that has aced every benchmark, and then ask the one question the score sheet never answered: when this machine enters a dangerous state in the field, is it detected, blocked, or exited fast enough to protect the person standing next to it? Deployment approval lives exactly at that boundary between learning and safety engineering, where "usually behaves well" is not the same claim as "safe to release."

As Figure 54.6.1 illustrates, deployment approval requires more than a successful demo: it needs a bounded operating domain, an evidence map, and named residual risks. The approval tuple is $$A = (\mathrm{ODD}, \mathrm{evidence}, \mathrm{defeaters}, \mathrm{residual\ risk}, \mathrm{rollback}),$$ where every element must be explicit enough for an external reviewer or internal safety board to challenge. Here ODD stands for Operating Design Domain: the bounded set of environmental and operational conditions within which the safety argument holds. Many teams then arrange this tuple in a goal-structuring tree using Goal Structuring Notation (GSN), a graphical notation, described in full later in this section's Library Shortcut, that links each top-level safety claim through subclaims to the specific evidence and challenge conditions that support it, so each top-level release claim has child claims, evidence links, and explicit challenge conditions. The rest of this section builds toward that structure: it first defines each tuple element, then walks a worked example through the release gate, and only then formalizes the gate as GSN.

This structure matters in embodied AI because a robot acting in the physical world cannot be patched mid-motion. A software bug in a web service causes a failed request; the same class of reasoning gap in a mobile manipulator causes an irreversible collision. A benchmark score is evidence that something worked under controlled conditions, not a promise that it will work in the world. Each tuple element corresponds to a distinct failure mode that has caused real incidents: missing ODD boundaries led to autonomous vehicles operating in unvalidated weather; missing defeaters meant teams did not anticipate sensor degradation scenarios; missing rollback plans meant operators had no safe action when a system began behaving unexpectedly.

Figure 54.6.2 shows this tuple as a sequential gate: each element must be populated and traceable before the gate clears, and one missing element blocks approval no matter how strong the rest. Teams fill every element before the release board convenes. The hazard analysis from section 54.3 supplies the ODD as hard numerical thresholds. Evidence is artifacts linked to specific claims, gathered from evaluation panels, monitor tests, and override trials, typically compiled into a trace matrix (a table mapping each release claim to the specific document, log, or test result that supports it). Each defeater is a counter-argument that would invalidate a claim, so the team either refutes it with data or promotes it to a residual risk with a named owner. Rollback names the exact trigger and the sequence that returns authority to a human or halts the system. The tuple is complete only when every element is filled and traceable.

Worked Example

A delivery robot may be approved for indoor daytime office routes with marked lanes and trained operators, but not for crowded public spaces or unstructured loading docks. Approval is domain-bounded by design.

approval = {
    "odd_defined": True,
    "evidence_complete": False,
    "override_tested": True,
    "residual_risk_named": True,
}
approval["release_ready"] = all(approval.values())
print(approval)

{'odd_defined': True, 'evidence_complete': False, 'override_tested': True, 'residual_risk_named': True, 'release_ready': False}

Expected output: The release is blocked because the evidence set is incomplete. That is exactly how a safety gate should behave: incomplete evidence is itself a meaningful decision signal.

Consider a concrete case. A last-mile delivery robot (Starship Technologies generation-3 chassis, indoor variant) was scoped to corridors at least 1.2 m wide, illuminance above 50 lux, and pedestrian density below 0.3 persons per square meter. Its ODD card set a maximum rated speed of 0.8 m/s and an emergency-stop latency budget of 120 ms. The release gate checked four evidence items: a 500-episode evaluation panel showing a hazard-intervention rate below 0.2 per hour, a monitor-coverage test confirming that 97% of injected faults triggered an alert within 80 ms, a human-override test across 40 scripted scenarios with zero missed stops, and a hazard log that closed all severity-1 items and deferred three severity-2 items to named owners. Monitor coverage missed the 99% target the safety case specified, so the evidence_complete flag stayed False and the team held the release for two weeks of sensor-fusion tuning before re-evaluation. That single gap, 97% coverage versus a 99% gate, added 14 days and three full re-runs of the 500-episode panel before approval cleared. One missed threshold turned a 500-episode evidence budget into a 1,500-episode one, tripling the evaluation cost before anyone touched the robot's code.

What would have happened if that 97% coverage figure had never been checked and the robot had shipped anyway? The question is not rhetorical: exactly this scenario, a system that passed every headline benchmark but missed a quiet coverage threshold, is the pattern behind most post-deployment safety incidents in fielded autonomous systems.

Avoiding that pattern is precisely what disciplined approval practice is built to do. Good approval practice names the boundary conditions in hard thresholds. A system can be approved for one environment, one workload, or one operator model without implying readiness everywhere else. In mature programs, each approval claim is backed by a trace matrix that points to benchmark artifacts, monitor tests, override results, and the exact owner for unresolved residual risks.

The gate as a constrained objective

Naming those boundary conditions and their owners in prose is the human-readable half of the gate; the other half is stating the same boundary as a constrained objective a machine can check. Formally, a release gate should check an objective such as task utility subject to explicit constraints on hazard rate, intervention latency, monitor coverage, and residual risk. The approved policy is therefore not merely the argmax of performance (the single candidate that scores highest on the benchmark), it is the member of the candidate set whose evidence satisfies the safety case under the frozen operating-domain assumptions.

In practice, one of the most dangerous approval failures is domain creep after the gates close, where a system slowly begins operating outside the bounded domain because no one wrote the exclusions sharply enough or monitored them after release.

Cross-References

This section sets up the capstone logic of Section 54.7 on assurance arguments and prepares the transition to Chapter 55 on deployment architecture. The approval tuple introduced here (ODD, evidence, defeaters, residual risk, rollback) becomes the leaf structure of the full assurance graph in Section 54.7, where each element is expanded into a claim, context, and evidence pairing that a release board can trace and challenge individually.