Section 15.3: Actor-critic and advantage estimation (GAE)

"The critic does not tell the actor what to do. It tells the actor whether what it did was better or worse than it deserved to expect, and that is enough."

A Careful Control Loop

Picture a walking robot that takes 200 actions, falls once near the end, and earns a single episode return: REINFORCE blames all 200 actions equally, including the good ones taken before the misstep. That coarse, all-or-nothing credit is exactly what makes plain policy gradients so noisy for embodied agents, where a late slip, contact bounce, or tracking error can dominate the whole episode return. Actor-critic methods sharpen the blame by learning \(V_\phi(s)\), a value function that estimates how much return is expected from a state before the next action is chosen. Figure 15.3A captures the division of labor: the actor decides what to do next, while the critic judges whether the last action beat the state's expectations.

Figure 15.3B traces the full loop: the actor acts, the environment responds, the critic scores the surprise as a TD residual, and GAE chains those residuals into the advantage that drives the next actor update. The actor update uses an advantage estimate:

$$A_t = Q(s_t,a_t)-V(s_t).$$

From advantage to TD residual

In practice, Proximal Policy Optimization (PPO) often starts from the temporal-difference residual:

$$\delta_t=r_t+\gamma V(s_{t+1})-V(s_t),$$

The TD residual matters for physical robots because a robot cannot wait until the episode ends before updating. Real hardware wears, batteries drain, and safety envelopes must hold mid-episode. The residual compares the immediate reward plus the discounted next-state value against the current-state prediction. This comparison measures local surprise: how much better or worse the next moment was than the critic expected. A positive residual means the transition was unexpectedly good; a negative residual flags an outcome worse than predicted. That per-step signal lets the critic update continuously from partial trajectories rather than only from complete episodes.

The TD residual \(\delta_t\) subtracts the critic's prediction \(V(s_t)\) from the one-step evidence \(r_t + \gamma V(s_{t+1})\). The bootstrap term \(\gamma V(s_{t+1})\) replaces real future returns with the critic's estimate, which keeps variance low. When the critic is accurate, \(\delta_t\) averages near zero, so the actor receives only genuine new information. GAE chains these residuals backward: a surprise at step \(t+k\) reaches earlier steps with weight \((\gamma\lambda)^k\).

Having defined \(\delta_t\), GAE combines a whole rollout of these residuals into a single advantage estimate by summing them with exponentially decaying weights:

$$\hat A_t^{\mathrm{GAE}(\gamma,\lambda)}=\sum_{l=0}^{\infty}(\gamma\lambda)^l\delta_{t+l}.$$

This sum is the section's core deliverable: it is how GAE actually turns per-step TD residuals into the advantage estimate that the actor uses, and the algorithm box, worked example, and lambda-sweep lab that follow all show this formula in use.

A critic that merely predicts reward is a scoreboard; a critic that predicts expected return from each state is a conscience, because it lets the actor know not just what happened but whether it deserved to happen.

Theory

The parameter \(\lambda\) controls a bias-variance tradeoff. When \(\lambda=0\), the advantage uses one-step TD evidence, which is low variance but depends heavily on critic accuracy. When \(\lambda\) is near 1, the estimate resembles Monte Carlo return, the sum of actual rewards observed for the rest of the episode with no bootstrapping from the critic, which uses longer evidence but becomes noisier. PPO commonly uses values near 0.95 because that often balances delayed reward evidence with manageable variance. The practical difference is large. In a standard MuJoCo hopper benchmark, REINFORCE typically needs roughly 50,000 episodes to reach a stable gait. Actor-critic GAE (\(\lambda=0.95\)) reaches the same policy quality in around 300 episodes, because the critic absorbs most of the episode-level noise before it reaches the gradient. Exact figures vary by environment and reward shaping, but the two-orders-of-magnitude gap is representative.

For embodied control, this tradeoff plays out in time. A one-step residual misses the stumble a foot placement caused three frames later; a long-horizon return blames that same placement for an unrelated collision after a perception glitch. GAE hands the builder one knob: how far credit travels backward through the rollout.

Worked Example

Before reading the code below, guess: in a four-step rollout where step 2 yields a reward of 1.0 and every other step yields zero, which earlier steps receive a positive advantage, and by roughly how much does the credit decay with each step backward?

Code Fragment 1 computes GAE for a short rollout. Notice that the advantages are computed backward because later residuals influence earlier credit.

# Compute Generalized Advantage Estimation from rewards and value predictions.
# The backward recursion sends delayed evidence to earlier sampled actions.
rewards = [0.0, 0.2, 1.0, -0.1]
values = [0.3, 0.4, 0.6, 0.2, 0.0]
gamma = 0.99
lam = 0.95

advantages = []
gae = 0.0
for t in reversed(range(len(rewards))):
    delta = rewards[t] + gamma * values[t + 1] - values[t]
    gae = delta + gamma * lam * gae
    advantages.insert(0, round(gae, 3))

print("advantages:", advantages)

This trace gives the mental model. Step 3 has a negative advantage because the outcome was worse than the critic expected. Earlier steps remain positive because the later reward still supplies evidence that those actions helped set up a useful state.

Practical Recipe

1. Train the actor and critic on the same rollout batch so advantages match the behavior policy.
2. Bootstrap from \(V(s_{t+1})\) when a rollout segment ends by truncation, but not when the episode truly terminates (see the Common Failure Mode callout below for why conflating the two is the most common GAE bug in practice).
3. Normalize advantages per batch to prevent reward-scale changes from dominating the policy loss.
4. Track explained variance, the fraction of the variance in the return targets \(G_t\) that the critic's predictions \(V_\phi(s_t)\) account for, where a value near 1 means the critic tracks returns well and a value near 0 means it predicts no better than the batch mean, or value error so a broken critic does not silently corrupt the actor update.
5. Audit GAE under perturbations, because delayed physical failures change how far credit should travel backward.

Answering that self-check correctly depends on keeping two bookkeeping streams apart, because the actor and critic draw on the same rollout yet pull in different directions. The actor and critic optimize different targets from the same experience, a principle called separate losses from shared rollouts. The actor asks which sampled actions should become more likely. The critic asks which states predict future return. If these targets are mixed casually, a bug in value learning can masquerade as a policy improvement.

Embodied systems add one more complication: the critic often sees partial observations, not the full physical state. If the value function cannot infer hidden contact state, object slip, battery sag, or actuator temperature, its baseline will be noisy. That does not make actor-critic invalid, but it does mean the value diagnostics must be read alongside physical failure labels.

Whichever estimator from the table above you settle on, its raw output still has to be rescaled before PPO can use it safely, which is where the next step comes in.

Code Fragment 2 shows the second piece most PPO implementations apply after GAE: advantage normalization. This is not part of the theorem; it is a practical stabilizer that keeps the policy loss scale consistent across batches.

1. Compute advantages before shuffling minibatches, using the rollout time order.
2. Normalize advantages after GAE, using the full batch mean and standard deviation.
3. Keep raw advantages in logs so reward-scale and critic failures remain visible.
4. Train the value function on returns compatible with the same bootstrap convention.
5. Plot value prediction, return target, and failure labels for several complete episodes.

# Normalize advantages after GAE so PPO sees a stable loss scale.
# Keep raw values for diagnostics because normalization can hide reward bugs.
advantages = [0.746, 0.691, 0.316, -0.300]
mean_advantage = sum(advantages) / len(advantages)
variance = sum((x - mean_advantage) ** 2 for x in advantages) / len(advantages)
std_advantage = variance ** 0.5

normalized = [(x - mean_advantage) / (std_advantage + 1e-8) for x in advantages]
print([round(x, 3) for x in normalized])

Normalization keeps the loss scale stable when everything works, but it also hides trouble when the critic underneath is sick, so debugging starts one level down. When actor-critic training fails, check critic health first. A value loss that falls while episode behavior worsens can mean the critic learned the wrong shortcut, such as predicting timeout length rather than task progress. A value loss that never falls can turn every advantage estimate into high-variance noise.