Section 7.3: PID control, intuition and tuning

"P reacts to now, I confesses to the past, D worries about what comes next. Together they argue until the error is small enough to ignore."

A PID Tuner at 2 AM

Three numbers, written down by hand in under a minute, decide whether a robot arm settles gently onto a cup or rings like a struck bell: that is the entire bargain of PID control. Learning to read those three gains by feel is the skill that matters here. The section defines the object of study, connects it to the agent loop, then tests it with a compact implementation.

The key question is concrete. On a Franka Emika Panda joint controller running at 1 kHz, the agent knows the commanded reference angle, observes the encoder position (and its filtered velocity), and the only action available is a single torque value clamped to the joint's limit (87 Nm on the large links, 12 Nm on the wrist). The evidence that tuning worked is a logged step-response trace, where a step response is the controller's reaction over time to an instantaneous jump in the reference command: rise time under 50 ms, overshoot below 5 percent, and zero steady-state drift after the integral term settles. Everything in this section reduces to making that one torque-per-millisecond decision trustworthy.

Theory

Consider a robotic arm holding its end-effector at a target joint angle. The motor driver needs a torque command every millisecond. A neural policy trained offline cannot react to an unexpected cable-friction spike in that loop; a full dynamics model is too slow to invert online. PID solves this exactly: it reads the current angle, computes how far off it is, and issues a corrective torque in under a microsecond, with no model needed. That is the role PID fills in embodied AI: fast, interpretable, actuator-level error correction that runs inside the loop while slower learned planners operate above it. Figure 7.3A sketches this controller as a block diagram, summing the proportional, integral, and derivative terms into a single torque command for joint position control.

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

PID control is useful because it gives three readable levers before a learned policy enters the loop. The proportional term reacts to the current error, the integral term removes persistent bias, and the derivative term resists fast changes that would otherwise overshoot. A tuning pass should change one lever at a time and record rise time, overshoot, settling time, steady-state error, saturation events, and noise sensitivity. The block diagram in Figure 7.3B below traces how these three terms combine: the error signal \(e(t)\) feeds all three terms in parallel, their outputs sum into a single command \(u(t)\), and the measured plant output feeds back to close the loop, where the plant is the physical system being controlled (here the robot joint and its actuator).

Why proportional alone is not enough

With the three parallel terms now wired together in the loop, the next question is why two of them, integral and derivative, are needed at all rather than just turning the proportional gain up; the answer begins with the bias that proportional control alone can never remove. Persistent bias matters in embodied AI because real actuators always carry unmodeled loads: a robot arm fights gravity at every pose, a mobile base experiences floor friction that differs from the simulator, and a soft gripper compresses against an object with force the model never captured. Proportional control alone reaches a steady state where the remaining error is just large enough to produce the torque that balances the load. For a representative joint gain in this range, that residual is roughly 0.5 degrees of joint error on a 1 kg arm held horizontal. On a 5 kg arm at the same gain, the residual scales with the load and typically grows to roughly 2.5 degrees of drift, enough to miss a cup by three centimeters at the wrist. That leftover error is not a tuning failure. Physics forces it structurally. It accumulates into position drift, grasp slip, or trajectory deviation. A learned policy above the loop cannot recover without reissuing corrective commands every cycle.

Before reading on, consider: why not raise the proportional gain until the residual error is negligible? Try it on any real actuator. The arm begins to oscillate, and the oscillation worsens with every increase. You are chasing a structural physics constraint with a lever that also destabilizes the system. That tradeoff is what the integral term was designed to escape.

The integral term eliminates this by summing error over time. Each control cycle adds the current error, multiplied by the timestep, to a running accumulator. The controller multiplies that accumulator by \(K_i\) and adds it to the command. As long as any error remains, the accumulator grows, which grows the command, which pushes the plant closer to the reference until the error reaches zero and the accumulator stops increasing. At that point the integral term is holding exactly the constant torque needed to balance the persistent load, freeing the proportional term to respond only to transient deviations.

Worked Example

PID has three classic failure modes, and each one is reproducible in a few lines on the same 1D mass. Code Fragment 7.3.1 exposes all three: integral windup (a large \(K_i\) keeps charging while the actuator saturates, then overshoots), derivative kick (a step change in the setpoint makes \(\dot e\) spike, so a derivative computed on the error fires a huge first command; Code Fragment 7.3.3 later walks through this arithmetic by hand), and derivative noise (differentiating a noisy measurement amplifies the noise). The fixes are an anti-windup clamp and computing the derivative on the measured output rather than on the error. Derivative noise itself is typically tamed with a first-order low-pass filter on the derivative term (for example, filtering \(de/dt\) at a cutoff well below the control rate) before multiplying by \(K_d\); this trades a small amount of phase lag for a large reduction in high-frequency chatter, and is the standard third guard alongside the anti-windup clamp and derivative-on-measurement shown below. The scale of the difference is not subtle: switching to derivative on measurement, not on error drops the first issued command from 260 to 10, a 26-fold reduction, before any other parameter changes.

import numpy as np

m, dt, T = 1.0, 0.02, 5.0
n = int(T / dt)
u_max = 6.0

def run(kp, ki, kd, anti_windup=True, deriv_on_meas=False, ref=1.0):
    x = v = ei = 0.0
    e_prev = y_prev = 0.0     # e_prev = 0 so the setpoint step shows the kick
    overshoot, first_cmd = 0.0, None
    for k in range(n):
        y = x                                  # measured position
        e = ref - y
        ei_t = ei + e * dt
        # Derivative on measurement avoids the spike when the setpoint jumps.
        de = -(y - y_prev) / dt if deriv_on_meas else (e - e_prev) / dt
        u_unsat = kp * e + ki * ei_t + kd * de
        u = float(np.clip(u_unsat, -u_max, u_max))
        if (not anti_windup) or (u == u_unsat):    # anti-windup clamp
            ei = ei_t
        if first_cmd is None:
            first_cmd = u_unsat
        x += v * dt; v += (u / m) * dt
        overshoot = max(overshoot, x - ref)
        e_prev, y_prev = e, y
    return x, overshoot, first_cmd

xf, ov, _ = run(10, 3, 5)
print(f"tuned         : final={xf:.3f}  overshoot={ov:.3f}")
_, ov_off, _ = run(10, 30, 5, anti_windup=False)
_, ov_on,  _ = run(10, 30, 5, anti_windup=True)
print(f"windup off/on : overshoot {ov_off:.3f} -> {ov_on:.3f} with anti-windup clamp")
*_, kick_e = run(10, 3, 5, deriv_on_meas=False)
*_, kick_m = run(10, 3, 5, deriv_on_meas=True)
print(f"setpoint step : first command {kick_e:.1f} (deriv on error) vs {kick_m:.1f} (deriv on measurement)")

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

Gains that look stable on a screen are only a hypothesis; the hardware step-response test is the verdict.

That same nested-loop diagnostic, scaled from a single quadrotor to an entire open-source autopilot, is exactly what production flight stacks automate.

Production Pattern

PID control, intuition and tuning sits inside the Part II robotics contract: geometry defines where things are, kinematics defines what motion is possible, dynamics defines what motion costs, control defines how errors are corrected, and sensing defines what the agent can know on time.

Tune PID by reading proportional, integral, and derivative effects separately before combining them. This makes the section useful to practitioners at every level: the idea has an intuitive role, a formal interface, a runnable check, and a failure mode that can be reproduced.

Use this recipe when turning PID control, intuition and tuning into code, a simulator experiment, or a robot diagnostic. The point is not to use every library. The point is to keep the hand-built baseline and the maintained-tool path comparable.

1. Write the control objective, measured state, actuator command, update rate, and saturation policy.
2. Run a step-response test before adding learning, with overshoot, settling time, and steady-state error logged.
3. Compare the hand controller with python-control, CasADi, Drake, do-mpc, or ROS 2 control on the same plant model.
4. Record latency, missed deadlines, saturation events, constraint violations, and recovery actions.
5. Only compare controllers and policies when they share sensors, action limits, disturbance tests, and safety checks.

A learned policy can hide an unstable PID inner loop until the disturbance changes. Check update rate, delay, actuator saturation, integral windup, derivative noise, and fallback behavior before scaling training. For this section, first reproduce one tiny PID update by hand, then rerun it through python-control or the robot controller stack. If the two disagree, inspect sign convention, sample time, derivative filtering, and anti-windup behavior before changing the model.

# Compute one PID update from current error, accumulated error, and error slope.
# The sign of the derivative term shows whether the controller is damping the motion.
error_prev = 0.6
error_now = 0.4
integral_error = 1.2
dt = 0.1
kp, ki, kd = 2.0, 0.5, 0.1

derivative = (error_now - error_prev) / dt
command = kp * error_now + ki * integral_error + kd * derivative
print(f"derivative={derivative:.1f}, command={command:.1f}")