Section 15.4: Trust regions; TRPO to PPO

"You may improve, but not too far from where you stood when you measured. The data remembers the policy that made it."

A Cautious Policy Interface

How can a single gradient step that looked perfect on yesterday's rollout topple a robot that was walking flawlessly an hour ago? A policy-gradient update can be too successful at following its own noisy estimate: one batch says a rare action looked good, the optimizer makes that action much more likely everywhere, and the next rollout discovers that the policy has stepped outside the region where the data were informative. In embodied agents, that can mean falls, collisions, or controllers driven into saturation.

A policy that cannot stay near the data that trained it is not a policy at work; it is a policy in free fall. As Figure 15.4A illustrates, the cure is to keep each update inside a safe trust region around the behavior that generated the evidence. TRPO frames the solution as a constrained optimization problem:

$$\max_\theta \mathbb{E}\left[\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)}\hat A_t\right]\quad\text{subject to}\quad \mathbb{E}\left[D_{\mathrm{KL}}(\pi_{\theta_{\mathrm{old}}}\Vert \pi_\theta)\right]\le \delta.$$

The probability ratio compares the new policy with the old behavior policy on the same sampled actions. The KL constraint says the new policy should not move too far from the old one in distribution space.

Why KL, and not any distance?

On a physical robot, violating this constraint has immediate mechanical consequences. A joint torque command drawn from a shifted distribution can exceed hardware limits within a single control cycle. It can also trip overcurrent protection (a safety circuit that cuts power when current draw exceeds a safe threshold) or excite resonant modes in the linkage (vibration frequencies at which the mechanical structure amplifies rather than damps an input, risking fatigue or loss of control). We choose KL divergence because it upper-bounds the expected performance difference between the two policies, not because it is the only mathematically valid notion of distance between distributions. That bound gives a theoretically grounded radius, not an arbitrary parameter.

Mechanically, \(D_{\mathrm{KL}}(\pi_{\mathrm{old}} \Vert \pi_\theta)\) sums the log ratio \(\log \frac{\pi_{\mathrm{old}}(a|s)}{\pi_\theta(a|s)}\) over all actions, weighting each by the old policy's probability. The sum is zero when the two distributions match. It grows as the new policy shifts probability mass to actions the old policy rarely chose. Constraining this sum to \(\delta\) stops the optimizer from reassigning large probability to rarely-sampled, poorly-evaluated actions.

Theory

TRPO buys that validity radius with an expensive second-order constrained solve (an update that uses local curvature information, not just the gradient, to decide both direction and step size); PPO keeps the same trust-region instinct but pays for it with a single cheap operation instead.

PPO replaces TRPO's constrained solver with clipping. Define the ratio

$$r_t(\theta)=\frac{\pi_\theta(a_t\mid s_t)}{\pi_{\theta_{\mathrm{old}}}(a_t\mid s_t)}.$$

The clipped surrogate objective is

$$L^{\mathrm{CLIP}}(\theta)=\mathbb{E}_t\left[\min\left(r_t(\theta)\hat A_t,\operatorname{clip}(r_t(\theta),1-\epsilon,1+\epsilon)\hat A_t\right)\right].$$

If the advantage is positive, PPO stops rewarding the update once the new policy makes the action much more likely than the old policy did. If the advantage is negative, PPO stops rewarding the update once the new policy makes the action much less likely. The clip does not enforce a hard KL limit, but it removes incentive for many destructive updates. Figure 15.4B plots this behavior for a positive-advantage sample: the objective rises linearly with the probability ratio until the upper clip boundary, then flattens.

The TRPO-to-PPO move is short to state. TRPO maximizes \(\mathbb{E}\left[\frac{\pi_\theta}{\pi_{\mathrm{old}}}\hat A\right]\) subject to \(D_{\mathrm{KL}} \le \delta\), solving it with a conjugate-gradient step (an iterative method that approximately solves the large linear system a second-order update requires, avoiding the cost of inverting the full curvature matrix) and a line search (a check that shrinks the proposed step until it actually satisfies the KL constraint) on a second-order model. PPO keeps the same surrogate ratio but swaps the hard constraint for the clipped objective above, so the constraint becomes a flat ceiling on the ratio that fits inside ordinary minibatch SGD. The substitution pays off in wall-clock: on a 24-core CPU cluster, TRPO's conjugate-gradient solve adds 40 to 60 seconds per update and hours over a full locomotion curriculum, while PPO's clip costs microseconds inside the backward pass that already runs.

Worked Example

To see how those two brakes actually engage, it helps to watch the clipped objective act on concrete ratios rather than reason about it in the abstract.

Code Fragment 1 calculates the unclipped and clipped PPO terms for a few rollout samples. The clipped objective blocks extra credit when the probability ratio moves outside the trust band.

# Compute PPO clipped surrogate terms for sampled ratios and advantages.
# The min operation removes incentive for updates beyond the trust band.
ratios = [0.72, 0.93, 1.08, 1.31]
advantages = [0.6, -0.4, 0.8, 0.5]
epsilon = 0.2

for ratio, advantage in zip(ratios, advantages):
    clipped_ratio = min(max(ratio, 1 - epsilon), 1 + epsilon)
    unclipped = ratio * advantage
    clipped = clipped_ratio * advantage
    objective_term = min(unclipped, clipped)
    print(ratio, advantage, "clip:", clipped_ratio, "term:", round(objective_term, 3))

The expected output shows two different PPO behaviors. Ratios inside the trust band, such as 1.08, keep their natural objective term, while an oversized ratio like 1.31 is cut back to the clip limit, preventing one minibatch sample from driving an excessively large policy update.

The first row shows a different subtlety. A positive-advantage action became less likely, so clipping does not rescue it. The objective term remains low because the new policy moved against the learning signal.

Practical Recipe

1. Save old log probabilities during rollout and compute ratios from new log probabilities during training.
2. Use clip ranges such as 0.1 to 0.3 as starting points, then tune with KL, entropy, and return stability.
3. Stop or shrink updates when approximate KL exceeds the target by a large margin.
4. Track the fraction of samples clipped, because a high clip fraction means many gradients are pushing against the trust band.
5. Watch embodied safety metrics during updates, not only episodic return.

TRPO and PPO are best understood as responses to the same failure: on-policy data become stale quickly. TRPO solves the problem more formally with a constrained update based on KL divergence. PPO accepts a less exact constraint because the clipped loss is simple, scalable, and easy to combine with minibatch stochastic gradient descent.

For embodied systems, the important diagnostic is not only the scalar return. A KL spike can precede visible behavior collapse by one update: the rollout that caused the update still looked good, while the next policy executes a new distribution of motions. Logging approximate KL, clip fraction, entropy, and action standard deviation gives the team early warnings before a robot-level failure becomes expensive.

Code Fragment 2 shows a compact KL and clip-fraction diagnostic. These numbers belong next to return curves because they explain whether PPO improved behavior by a controlled update or by a risky jump.

1. Compute ratios as exp(new_log_prob - old_log_prob), not by dividing rounded probabilities.
2. Report approximate KL for each update epoch, not only at the end of training.
3. Stop the epoch loop early when KL exceeds the target threshold.
4. Pair every return plot with entropy, KL, and clip fraction.
5. Inspect videos or state traces from the update after the largest KL movement.

# Compute PPO diagnostics from old and new log probabilities.
# KL and clip fraction tell you whether the update stayed near the rollout policy.
import math

old_log_probs = [-0.9, -0.4, -1.2, -0.7]
new_log_probs = [-0.7, -0.5, -0.8, -0.2]
epsilon = 0.2

log_ratios = [new - old for old, new in zip(old_log_probs, new_log_probs)]
ratios = [math.exp(log_ratio) for log_ratio in log_ratios]
approx_kl = sum((ratio - 1.0) - log_ratio for ratio, log_ratio in zip(ratios, log_ratios)) / len(ratios)
clip_fraction = sum(abs(r - 1.0) > epsilon for r in ratios) / len(ratios)

print("ratios:", [round(r, 3) for r in ratios])
print("approx_kl:", round(approx_kl, 3))
print("clip_fraction:", round(clip_fraction, 2))

The expected output means three of the four sampled ratios are already outside the nominal PPO trust region, which is why the clip fraction rises to 0.75. A KL of 0.067 in such a tiny example should be read as a warning that additional epochs would likely over-update the policy.

When PPO collapses after a promising update, inspect the trust-region diagnostics before changing the reward. A high clip fraction suggests the learning rate, epoch count, or advantage scale is too aggressive. A low entropy trace suggests the policy lost exploration. A KL spike suggests old rollout data were overused.