Section 19.3: Safe exploration

"Exploration is permitted; spending the safety margin to find out what happens is not."

A Cautious Policy Interface

The constrained reward formulation here extends the reward specification from Chapter 18; the cost-gated probe model and cross-chapter connections are introduced in section 19.1.

How do you let a robot try something it has never done, on real hardware, while guaranteeing it can always walk the action back? The object of study here is not only the reward-seeking policy, but the safety envelope that decides which exploratory actions are admissible. Figure 19.3B traces the full pipeline that the rest of this section builds out: a proposed action passes through an admissibility gate, executes only if it clears, and feeds reward and cost back to a monitor that adjusts the constraint.

The key question is practical: what constraint must never be violated, which near-violation should trigger intervention, and how does the evaluation report reward and cost together?

Theory

We can view the agent at time \(t\) as receiving an observation \(o_t\), maintaining an internal state estimate \(\hat s_t\), proposing an action \(a_t\), and passing that action through a constraint check before execution. A constrained objective tracks both reward \(R\) and cost \(C\), then requires a condition such as \(E[C] \le d\) for a chosen limit \(d\).

Make the safety channel explicit for the specific sensing modality in use. On a Franka Panda arm, clearance comes from the joint-torque sensor stream (1 kHz). A latency spike above 5 ms in that stream is itself a cost event: the shield can no longer guarantee that a proposed motion stays within the 10 N·m safety torque limit before the next control cycle. On a Boston Dynamics Spot navigating an unstructured warehouse floor, the relevant signal is the depth-camera point cloud (typically 30 Hz at 1280x720). The gate treats a clearance estimate older than 100 ms as expired and holds the action until a fresh reading arrives. In both cases the safety channel records four fields: the sensor timestamp, the clearance value, the intervention type (hold, redirect, or emergency stop), and whether recovery succeeded within the episode. "Safe" without these fields is a label, not a testable property.

Worked Example

Code Fragment 19.3.1 implements a tiny action shield. The policy proposes exploratory actions, but the shield vetoes any action whose clearance is below the limit or whose recovery flag is false.

# Filter exploratory actions through a simple safety shield.
# The chosen action must be informative and still recoverable.
candidates = [
    {"action": "inspect shelf gap", "value": 0.70, "clearance": 0.22, "recoverable": True},
    {"action": "squeeze behind shelf", "value": 0.95, "clearance": 0.07, "recoverable": False},
    {"action": "rotate camera", "value": 0.50, "clearance": 0.40, "recoverable": True},
]

safe = []
for item in candidates:
    allowed = item["clearance"] >= 0.15 and item["recoverable"]
    print(item["action"], "allowed" if allowed else "vetoed")
    if allowed:
        safe.append(item)

print("selected", max(safe, key=lambda item: item["value"])["action"])

Expected output: the printed trace should show both allowed and vetoed actions. If the run reports only reward, it cannot prove that exploration respected the constraint.

With the shield understood from scratch and the library tooling in place, the next task is to turn those pieces into a repeatable workflow that a team can run on every checkpoint.

Practical Recipe

1. Write the reward metric and cost metric before choosing a model.
2. State the hard constraint, soft constraint, intervention rule, and recovery condition separately.
3. Build a shielded baseline that is simple enough to debug by inspection.
4. Record failures as structured cases: constraint miss, false veto, delayed intervention, unsafe recovery, or evaluation mismatch.
5. Run at least one perturbation test that pushes the policy near the safety boundary.

Consider a specific case: the Safety Gymnasium benchmark "SafetyPointGoal1" trains a point-mass robot to reach goals while keeping cost (contact with hazard circles) below a budget of 25 per episode. Constrained Policy Optimization (Achiam et al., 2017) on this task typically achieves roughly 25 reward units while holding mean episode cost near the budget, whereas unconstrained Proximal Policy Optimization (PPO) commonly reaches 28 reward units but incurs 60+ cost events per episode; exact figures vary with seed, network size, and library version, so treat these as representative rather than guaranteed. The difference shows that tracking cost separately is not a formality: an agent optimizing reward alone discovers that cutting through hazard zones is a shortcut, and the cost metric is the only signal that catches this.

Safe exploration becomes useful when it is tied to a closed-loop safety contract. The contract names the observation stream, the state estimate, the action representation, the cost signal, the intervention mechanism, and the evaluation artifact. Without that contract, a model can look capable in a notebook while violating a constraint that nobody logged.

The graduate-level habit is to separate four claims. The reward claim explains what the policy tries to accomplish. The cost claim explains what must stay bounded. The shield claim explains which actions are modified before execution, and this filter is often called the admissibility gate, the single decision point that separates curiosity from damage. The evidence claim records which measurements would convince a skeptical builder that safety held during exploration.

Why The Gate Runs Before Execution

The admissibility gate matters in physical systems because a robot cannot undo a fallen shelf, a broken joint, or a startled worker. Simulation allows infinite retries; hardware does not. Without a pre-execution filter, a single high-value exploratory action can exhaust the cost budget, trigger an emergency stop, or damage the robot through irreversible contact before the policy learns anything from the outcome. Teams running MuJoCo contact experiments often report a sharp gap, though the exact ratio depends heavily on task and reward shaping. Unconstrained PPO can need on the order of 40,000 episodes to reach a competent policy. The same task with an admissibility gate has been reported to converge in around 800, because the gate removes the long tail of recovery-from-damage episodes. Those episodes dominate training time yet contribute no usable gradient signal.

The gate evaluates two quantities for each candidate action before issuing any motor command. First, a clearance score measures the distance from the nearest constraint boundary (physical, regulatory, or torque-based). Second, a recoverability flag confirms that at least one safe return path exists from the resulting state. Only actions that pass both checks enter the executable set. The gate runs inside one control cycle, so engineers implement it as a fast deterministic filter rather than a learned module. This keeps its failure mode visible and auditable.

The evidence claim records which measurements would convince a skeptical builder that safety held during exploration.

What happens when the robot's clearance sensor lags by 200 ms and the admissibility gate approves an action that is already unsafe by the time the motor command fires?

Catching that failure before it reaches hardware is exactly why the gate and its logging depend on the right tooling, so the table below maps each part of the safety contract to a library that supports it.

A robust implementation starts with a tiny, inspectable safety wrapper, then moves to a maintained learner. The baseline logs reward, cost, proposed action, executed action, veto reason, intervention timing, and recovery outcome. The library version must emit the same schema, so the two are compared on one task rather than stitched together from separate experiments.

1. Write a one-paragraph safety contract with reward, cost, constraint, intervention, recovery, and failure fields.
2. Start with the smallest simulator or wrapper that exposes the constraint faithfully.
3. Run one deterministic smoke test and one near-boundary perturbation before scaling.
4. Save a single result artifact containing configuration, seed, reward, cost, interventions, traces, and failure labels.
5. Compare methods only when one script evaluates reward and cost on the same task panel.

Even a faithfully logged run will sometimes break the constraint, so the final skill is reading those same artifacts to locate where in the pipeline the safety guarantee slipped.

When safe exploration fails, avoid labeling the whole method as weak. First assign the failure to hazard sensing, constraint definition, false veto, late intervention, unsafe recovery, controller saturation, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause.