Section 54.5: Human override and safety testing

An override is complete only when the robot reaches a verified safe state.

A Safety-Critical Controls Researcher

Why This Matters

An operator hits the emergency stop and the robot arm keeps swinging for two more seconds: in that interval, is your system safe, or merely commanded to be? Human override and safety testing lives in exactly that gap, at the boundary between learning and safety engineering. The question is not whether the policy usually behaves well, but whether dangerous states are detected, blocked, or exited fast enough to protect people, equipment, and mission goals.

One useful statistic is the mean time to intervention (MTTI) $$\mathrm{MTTI} = \frac{1}{N}\sum_{i=1}^{N}(t_i^{override} - t_i^{hazard}),$$ paired with a success-after-override rate. Override quality is about both speed and whether the system enters a truly safe state afterward. Figure 54.5.2 lays out the full five-link chain this metric lives inside, showing that MTTI covers only the span from hazard onset to override command (the first three links) while the safety guarantee holds only at the very end.

Worked Example

A teleoperated humanoid may have an emergency stop button, but if the operator cannot tell which control mode is active or whether the button was accepted, the mitigation is weaker than it appears in a requirements sheet.

events = [
    {"hazard_s": 10.2, "override_s": 11.1, "safe_state_s": 11.9},
    {"hazard_s": 22.5, "override_s": 24.0, "safe_state_s": 25.8},
]
metrics = []
for e in events:
    metrics.append({
        "override_delay_s": round(e["override_s"] - e["hazard_s"], 2),
        "safe_state_delay_s": round(e["safe_state_s"] - e["hazard_s"], 2),
    })
print(metrics)

[{'override_delay_s': 0.9, 'safe_state_delay_s': 1.7}, {'override_delay_s': 1.5, 'safe_state_delay_s': 3.3}]

Expected output: The second event reaches safe state much later even though override still occurred. That distinction matters because override authority is only half the story; the platform must also settle safely.

Safety testing should include degraded sensing, workload, ambiguous alerts, and repeated interventions. Otherwise the operator interface may look robust only because the test removed the stress that makes it fail.

The test artifact is an override timing budget with detection time, communication delay, controller acceptance, actuator response, and final safe state. It is the difference between an emergency-stop button and an emergency-stop system.

That timing budget assumes a single, unambiguous chain of command; the moment more than one human can intervene, the budget is only as good as the rule that decides whose command wins.

What happens when two operators both press stop at the same moment and the controller has no rule for resolving them?

An authority matrix matters in embodied AI because a physical robot cannot pause while humans negotiate who is in charge. When multiple humans share control of one platform, the arbitration question connects directly to the shared autonomy models that govern how authority is divided between operators. A worker on a factory floor and a remote supervisor can press stop at the same instant and send conflicting commands. Without a defined priority order, the controller may lock, delay, or execute only one signal while it silently drops the other. In safety-critical robotics, that ambiguity has the same consequence as having no override at all. An authority matrix is a table. It assigns every actor (local operator, remote supervisor, automated watchdog) a numeric priority level, the interface it commands through, and the conditions under which its authority is active. The controller executes the highest-priority active command first. It rejects lower-priority signals until the higher-priority actor releases control or a timeout expires. Testing verifies that the arbitration logic holds under concurrent inputs, not just sequential ones.

A common failure is to measure emergency-stop latency but not verify the achieved state. Some platforms accept the override command quickly yet continue coasting, swinging, or drifting long enough to remain unsafe.

Cross-References

This section supports Section 54.6 on deployment approval and Section 54.7 on assurance cases, because override evidence often becomes part of the release dossier.

These distinctions between command and achieved state are not academic. The 2018 Uber ATG fatality in Tempe, Arizona shows what happens when the earliest link fails silently: the vehicle detected the pedestrian 6 seconds before impact but suppressed the alert because object classification was uncertain. A safety operator was present, yet the alert never reached them. The NTSB investigation blamed the hazard-detection-to-alert link, not the physical stop capability. MTTI measurements must be designed to catch exactly this pattern, because the delay between a hazard becoming detectable and an operator becoming aware is often larger, and harder to measure, than the delay between decision and actuator response. Here the actuator-response link would have taken roughly 0.3 seconds; the suppressed-alert link consumed the full 6 seconds, twenty times longer, yet alert latency rarely appears in override timing budgets at all.