Section 7.7: Controllers vs. policies; when learning helps and when it makes control unsafe

"A controller is built to be safe by design. A policy is trained to be useful by experience. Knowing which guarantee you need tells you which tool to reach for."

Section 7.7

Two pieces of software command the same robot arm: one can prove on paper that it will never breach a safety limit, the other has simply never breached one in ten million simulated trials, and only one of those statements survives contact with a friction value the simulator got wrong. That single asymmetry, between a guarantee that holds by construction and a track record that holds only on the training distribution, is the whole subject of this section. Figure 7.7A contrasts the two paradigms at a glance: the proved-safe controller and the trained-to-be-useful policy. First we define each object, then we connect it to the agent loop, then we test the combination with a compact implementation.

The choice between a controller and a policy turns on four practical questions: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Why does this distinction matter? You derive a classical controller from a model. You write down the dynamics, choose a control law, and prove (or at least verify empirically) that the closed-loop system is stable. The safety guarantee is a mathematical property of the design, not an empirical observation about past episodes. A learned policy comes from data instead: the optimizer finds parameters that maximize expected return across training scenarios. The resulting behavior can be extraordinarily capable, but the guarantee is statistical, not structural. If a disturbance or sensor reading falls outside the training distribution, the policy has no mechanism to fall back on first principles. Consider what that gap looks like in practice: a PID joint controller can be proven stable with a two-line Lyapunov argument and will hold that guarantee across 10 million timesteps; a policy trained for 10 million steps in simulation may fail on the very first real-world contact because the sim friction model was off by 15%. To put the asymmetry in concrete terms: correcting that friction mismatch in a classical model requires editing one parameter and re-running a 2-line stability check; correcting it in a learned policy typically requires 50,000 additional training episodes to wash out the old behavior. This is the proof gap between math and data, and it is the central design constraint this section addresses. A policy that works in simulation but fails on hardware is not a policy: it is a hypothesis that the real world has not yet tested. Understanding this asymmetry is the prerequisite for every design choice in this section.

Theory

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

A classical controller is usually strongest when the goal, state, model class, and safety limits are explicit. A learned policy is strongest when perception, contact variation, human preference, or high-dimensional context is too complex to hand-code. Consider a mobile manipulator performing household pick-and-place: a PID position controller can hold a joint angle to within 0.1 degrees reliably, but deciding which cup to grasp given cluttered visual context is where a learned vision-language policy earns its place. The safest hybrid treats the learned policy as a proposal generator and the controller or safety filter as the executable contract.

Worked Example

Having argued in theory that classical control should own the invariance guarantee while learning supplies the candidate action, we now make that division concrete in code. The safest place for learning in a control loop is as a residual on top of a classical controller, wrapped by a safety filter. The command is \(u = u_\text{base}(x) + \pi_\theta(x)\): the base controller owns the nominal behavior, and the learned residual nudges it. On a physical robot, this split matters because the base controller's stability guarantee survives even if the residual produces garbage: if \(\pi_\theta\) outputs zero (which happens under distribution shift), the robot still tracks its reference rather than freezing or oscillating toward a joint limit. An end-to-end policy has no such fallback; every output is load-bearing. The residual also bounds the damage a poorly trained policy can do, since actuator saturation limits \(|\pi_\theta|\) to a small fraction of the total authority. Mechanically, the loop evaluates the base controller first from the current state estimate, producing \(u_{\text{base}}\). The learned network then receives the same state and outputs a correction. The loop sums the two signals before the safety filter, so the filter sees a single command and enforces one consistent constraint. Training targets only the residual: the loss measures task error above the baseline, which keeps the learned component small and focused on the gap the classical law cannot close. A control barrier function (CBF) then enforces a hard safety set. For a barrier \(h(x)\ge 0\) (here, staying left of a wall), the filter requires \(\dot h(x) + \alpha h(x) \ge 0\) and projects the requested command onto the closest admissible one. Code Fragment 7.7.1 runs the same residual policy with and without the filter.

import numpy as np

# u = u_base(x) + pi_theta(x), guarded by a control barrier function.
# Plant: 1D mass. Barrier h(x) = x_wall - x >= 0  (stay left of the wall).
m, dt, x_wall, alpha = 1.0, 0.05, 1.0, 4.0

def u_base(x, v):   return 8.0 * (0.9 - x) - 4.0 * v   # nominal PD toward x=0.9
def pi_theta(x, v): return 5.0                          # learned residual (unsafe alone)

def cbf_filter(u, x, v):
    # h = x_wall - x, hdot = -v.  Enforce hddot + 2*alpha*hdot + alpha^2*h >= 0
    # with acceleration a = u/m affecting hddot = -a.
    h, hdot, a = (x_wall - x), -v, u / m
    margin = (-a) + 2 * alpha * hdot + alpha ** 2 * h
    if margin >= 0:
        return u, False
    u_safe = m * (2 * alpha * hdot + alpha ** 2 * h)    # minimal correction: margin -> 0
    return u_safe, True

for label, use_filter in [("no safety filter", False), ("with CBF filter", True)]:
    x = v = 0.0; interventions = 0; xmax = -9.0
    for _ in range(200):
        u = u_base(x, v) + pi_theta(x, v)
        if use_filter:
            u, did = cbf_filter(u, x, v); interventions += int(did)
        x += v * dt; v += (u / m) * dt
        xmax = max(xmax, x)
    breached = "BREACHED" if xmax > x_wall + 1e-3 else "safe"
    print(f"{label:>17}: max x={xmax:.3f} wall={x_wall} -> {breached}  interventions={interventions}")

Practical Recipe

The worked example showed why the learned and classical layers must stay separable; the steps below turn that principle into a deployment checklist, beginning with the interface decisions that determine whether the separation survives contact with real hardware.

1. Pin the observation space before choosing a policy architecture: on a Franka Panda, that means 7 joint positions, 7 joint velocities, and 6-axis end-effector wrench at 1 kHz, not a generic "state vector." On a Boston Dynamics Spot, add IMU at 400 Hz and four contact booleans at 333 Hz. Missing one sensor modality is the most common cause of policy rollout failures that never appeared in MuJoCo simulation.
2. Build a classical baseline first: a PD joint controller with hand-tuned gains is simple enough to verify analytically, and its step-response trace (overshoot under 5%, settling under 0.3 s for typical arm joints) gives you the performance floor the learned policy must beat.
3. Identify the sim-to-real gap sources before adding a learned residual: the three dominant gaps for contact-rich manipulation are surface friction (MuJoCo default 1.0 vs. real rubber-on-steel ~0.6), joint damping (often 10-30% off in URDF models), and sensor latency (real encoders add 1-5 ms; real cameras add 33-100 ms that sim skips). Model each gap explicitly or the residual policy will silently compensate for the wrong thing.
4. Record failures by physical cause, not software layer: distinguish "policy commanded torque above actuator limit" (action interface failure), "depth camera returned NaN at specular surface" (perception failure), and "CBF filter intervened on more than 30% of steps" (policy-safety mismatch) from each other. Mixing them into a single "task failed" label hides which part of the stack broke.
5. Run at least one out-of-distribution perturbation before declaring the policy ready: add 5 mm of random joint noise, swap the object mass by 50%, or change lighting to produce reflections the training data (for example, Open X-Embodiment or RT-X) did not cover. A policy that survives these tests under the same CBF filter is a meaningful result; one that only passes the nominal rollout is not.

Production Pattern

Where this sits in the Part II contract

Controllers vs. policies; when learning helps and when it makes control unsafe sits inside the Part II robotics contract: geometry defines where things are, kinematics defines what motion is possible, dynamics defines what motion costs, control defines how errors are corrected, and sensing defines what the agent can know on time.

Compare controllers and learned policies only under identical sensors, action limits, disturbances, and safety filters. Held to that standard, the comparison serves practitioners and researchers alike: each method carries an intuitive role, a formal interface, a runnable check, and a reproducible failure mode.

Use this recipe when turning Controllers vs. policies; when learning helps and when it makes control unsafe into code, a simulator experiment, or a robot diagnostic. The point is not to use every library. The point is to keep the hand-built baseline and the maintained-tool path comparable.

1. Write the control objective, measured state, actuator command, update rate, and saturation policy.
2. Run a step-response test before adding learning, with overshoot, settling time, and steady-state error logged.
3. Compare the hand controller with python-control, CasADi, Drake, do-mpc, or ROS 2 control on the same plant model.
4. Record latency, missed deadlines, saturation events, constraint violations, and recovery actions.
5. Only compare controllers and policies when they share sensors, action limits, disturbance tests, and safety checks.

A learned policy can hide an unsafe control interface until the disturbance changes. Check action limits, latency, recovery authority, safety-filter intervention rate, out-of-distribution observations, and fallback behavior before scaling training. For this section, first reproduce one controller-only case and one policy-proposal case under the same disturbance panel. If the two disagree, inspect whether learning improved perception or task selection, or merely bypassed a constraint that the classical controller was enforcing.