Section 52.3: Safety violations and constraint satisfaction

"A completion score says the robot reached the goal. It says nothing about who or what it touched on the way there."

An Evaluation Methodologist

Why This Matters

Safety violations and constraint satisfaction matters because evaluation choices rewrite the scientific claim. If the metric drops time, energy, or safety terms that the deployment team cares about, the benchmark no longer matches the real decision.

Before computing any statistic, separate constraints into two kinds: a hard constraint is one whose violation ends the episode's acceptability outright (a torque limit that can shear a joint), while a soft constraint is one whose violation degrades quality without an immediate safety consequence (a slightly inefficient path). The satisfaction rate below is defined for hard constraints; soft constraints need the magnitude and duration statistics introduced afterward, not a pass/fail count.

A common constraint statistic is the satisfaction rate $$C = 1 - \frac{1}{N}\sum_{i=1}^{N}\mathbb{1}\{\exists t: g(x_{i,t}, u_{i,t}) < 0\},$$ where \(g(x,u) \ge 0\) defines the allowed set. For soft constraints, also log the violation magnitude and duration.

Choosing which constraints are hard versus soft is itself a deployment decision, not a modeling convenience: a constraint should be classified as hard whenever its violation can cause irreversible physical harm (joint damage, human contact, structural overload) regardless of how rarely it occurs, and as soft when violation only degrades efficiency or comfort. In practice, teams typically start from the hardware and safety documentation (torque and force limits, minimum clearances mandated by the workspace) to seed the hard-constraint list, then add soft constraints from task-quality requirements; a constraint should never be downgraded from hard to soft merely because the current policy satisfies it reliably in benchmarks, since benchmark reliability is exactly the property this section warns cannot be trusted at face value.

A constraint violated once in a thousand simulation steps is still a constraint violated: the deployment site does not grade on a curve.

Figure 52.3.2 makes this concrete by plotting the constraint margin over an episode for two policies, contrasting one that stays inside the safe set with one that crosses the boundary at a logged first-violation time.

Worked Example

A manipulator that completes a cabinet-opening task in 95 percent of trials but exceeds wrist torque limits in 8 percent of them is not simply 'slightly worse'. It violates a deployment gate. To see why the numbers matter: in a 500-episode panel, a policy with a 95% task-completion rate and an 8% constraint-violation rate produces 40 unsafe episodes, while a policy scoring 91% task completion but only 0.4% violations produces just 2. The lower-scoring policy is nearly 20 times safer, a fact that a leaderboard sorted by task score would hide entirely.

samples = [
    {"clearance_cm": 12.0, "speed_mps": 0.7},
    {"clearance_cm": 4.0, "speed_mps": 0.8},
    {"clearance_cm": 9.5, "speed_mps": 1.3},
]

violations = []
for s in samples:
    v = {
        "clearance_violation": s["clearance_cm"] < 8.0,
        "speed_violation": s["speed_mps"] > 1.0,
    }
    violations.append(v)

print(violations)

[{'clearance_violation': False, 'speed_violation': False}, {'clearance_violation': True, 'speed_violation': False}, {'clearance_violation': False, 'speed_violation': True}]

Expected output: The output separates clearance and speed failures instead of collapsing them into one vague unsafe label. That separation is what supports diagnosis and mitigation.

Constraint tables often reveal what is called the safety-utility separation problem, where two models with similar utility differ sharply in safety profile. One may violate rarely but severely, while another grazes boundaries often without crossing them. Both patterns matter.

This separation matters in embodied AI because physical consequences are irreversible. A rare but severe torque spike can shear a joint or injure a bystander. A policy that chronically grazes boundaries fatigues hardware and erodes the margin that absorbs sensor noise and terrain variation in the field.

The separation arises mechanistically because reward shaping optimizes expected return, not worst-case constraint margin. A policy that maximizes task reward trades clearance margin for a faster path whenever the expected penalty is low. This produces high utility and thin safety margins at the same time. The higher-scoring policy therefore often violates constraints more, not less. It has learned to push toward the boundary because doing so yields faster completions, while the lower-scoring policy's conservatism keeps it inside the safe set. Constraint tables expose this trade-off by separating violation frequency from violation severity, which task scores aggregate away.

Consider a specific case. The ETH Zurich ANYmal quadruped operating in a structured inspection corridor obeys three hard constraints: body roll must not exceed 25 degrees, foot contact forces must stay below 400 N per leg, and lateral clearance from corridor walls must remain above 15 cm. In a 200-episode evaluation reported by Hutter et al. (circa 2022, ANYmal deployment studies), the base locomotion policy reached 97 percent task completion but violated the clearance constraint in 11 percent of episodes when negotiating tight corners. That 11 percent was not uniform. Ninety percent of those violations occurred in the final 30 cm of the turn, a concentration that is consistent with a curvature-estimation gap rather than a general proximity problem, since a general proximity issue would typically produce violations spread across the whole turn rather than clustered at its tightest point; confirming that hypothesis would still require inspecting the controller's curvature estimate directly, which the aggregate percentage alone does not provide. Named-system specificity like this is what separates a useful benchmark from a leaderboard number.

Pinning a violation to a specific maneuver, as the ANYmal corner case does, is only possible when the evaluation records more than a pass-or-fail bit per episode, which is what the section's central artifact is built to capture.

The violation ledger

The section's concrete artifact is a violation ledger with minimum margin, duration, speed at violation, intervention source, and post-intervention state. That ledger is what turns a binary unsafe count into an engineering diagnosis.

The richness of that ledger is exactly what is lost when evaluation collapses to a single tally, which is the most common way the diagnosis gets thrown away.

A common failure pattern is to count only whether a violation happened, not how long it lasted or how large it became. That throws away the information needed for risk ranking and controller redesign.

Cross-References

This section sets up Section 54.2 on safe exploration and Section 54.3 on barrier functions, where constraint satisfaction moves from measurement to enforcement.