Section 15.1: Direct policy optimization; stochastic policies

"A deterministic policy commits to one answer. A stochastic policy keeps a door open, and for a robot feeling its way through contact, that door is load-bearing."

A Policy Learning to Touch

Ask a value-based agent to grasp a bolt and it answers a question about the world: which action looks best right now? Ask a policy-gradient agent the same thing and it answers a question about itself. It asks how to bend the dial that governs its own behavior so the next thousand grasps land more often. That second question, as Figure 15.1A illustrates, is what lets a stochastic policy turn each exploratory action into reusable evidence by recording what it sampled, how likely that sample was, and what reward followed. Value-based methods ask, "Which action has the highest estimated value?" Policy-gradient methods ask a different question: "How should the parameters of the action distribution move so future sampled behavior becomes more successful?" For a Franka Panda arm sending 7-DoF joint velocity commands at 1 kHz, that shift matters enormously. A Q-table over continuous joint-velocity space requires enumerating combinations the hardware never reaches. A Gaussian policy with a learned mean and diagonal covariance gives the optimizer a differentiable handle on every dimension at once. That is what it means to optimize the distribution, not the decision. The same logic applies to Boston Dynamics Spot executing footstep offsets over uneven terrain. Its action space is a 12-dimensional continuous vector. Hardware joint limits force every sampled command through an actuator clip before it reaches the real servo. The policy must account for that clip explicitly; otherwise the gradient update credits the wrong action. Figure 15.1B below traces this loop step by step: observation into policy network, sampled action with its recorded log probability, environment reward, and the gradient update that flows back through that log probability rather than through the physics.

The object we optimize is the expected discounted return:

$$J(\theta)=\mathbb{E}_{\tau\sim\pi_\theta}\left[\sum_{t=0}^{T-1}\gamma^t r_t\right].$$

Here \(\theta\) are policy parameters, \(\tau\) is a trajectory of observations, actions, and rewards, \(\pi_\theta(a_t \mid o_t)\) is the probability or density assigned to an action, and \(\gamma\) discounts delayed consequences. Direct optimization means we adjust \(\theta\) to increase \(J(\theta)\) rather than first learning a separate action-value function and acting greedily from it.

Why the return, not the value table

For a physical robot, \(J(\theta)\) captures something value tables cannot: the compounding cost of sequential decisions under actuator noise and contact uncertainty. A grip that succeeds only when ten earlier joint adjustments were correct is worth more than ten isolated single-step rewards; the discount \(\gamma\) encodes exactly that temporal credit, preventing the policy from trading a stable grasp now for a marginally better one three seconds later that hardware jitter will likely forfeit.

To maximize \(J(\theta)\), we estimate its gradient from sampled trajectories. The environment physics are not differentiable, so the gradient cannot flow through the world; it flows through the policy's own log probabilities instead. Each rollout is a noisy gradient sample, and averaging many rollouts cancels the noise. The optimizer then steps in the direction that makes high-return trajectories more probable under the current distribution.

Theory

The policy is a probability distribution over the action space, not a lookup table. For a discrete mobile robot action set, \(\pi_\theta\) might be a softmax over forward, left, right, and stop. For a manipulation controller, it might be a Gaussian over joint velocity commands with a learned mean and standard deviation. In both cases, the policy must expose two things for training: the sampled action and the log probability of that exact action.

The central intuition is credit assignment through sampling (credit assignment means tracing an eventual reward back to the specific earlier action responsible for it). If a robot nudges a drawer handle and the drawer opens later, the update cannot differentiate through the drawer physics, contact impulses, camera exposure, or environment reset. It can still differentiate through the policy's own probability of the sampled motion. REINFORCE and PPO both stand on that foothold. A value-table approach on a 12-DoF arm joint-velocity space may need orders of magnitude more successful grip episodes before the Q-values stabilize than a Gaussian policy gradient needs on the same task. The policy gradient wins here because it updates a continuous distribution rather than filling a discrete table cell by cell, though exact ratios depend heavily on reward shaping and environment resets. In practice, coarsely discretizing a 12-DoF joint-velocity space to just 5 bins per dimension produces \(5^{12} \approx 244\) million table cells: a policy gradient sidesteps that enumeration entirely, updating a handful of network parameters instead, which is typically why even a naïve REINFORCE run can learn a stable grasp policy in as few as a few thousand episodes on simple tasks, whereas a Q-table variant may never fill enough cells to converge at all (exact episode counts depend heavily on task difficulty and reward shaping).

Worked Example

Code Fragment 1 below shows the smallest useful contrast: a deterministic controller always chooses the largest probability action, while a stochastic controller samples from the whole distribution and records the probability of what it did.

# Compare greedy action selection with stochastic sampling.
# The sampled action keeps a log probability, which policy gradients need.
import math
import random

random.seed(7)
actions = ["move_left", "move_right", "stop"]
logits = [0.2, 1.4, -0.7]

exp_logits = [math.exp(x) for x in logits]
total = sum(exp_logits)
probs = [x / total for x in exp_logits]

greedy_action = actions[probs.index(max(probs))]
sampled_action = random.choices(actions, weights=probs, k=1)[0]
sampled_prob = probs[actions.index(sampled_action)]

print("policy probabilities:", dict(zip(actions, [round(p, 3) for p in probs])))
print("greedy action:", greedy_action)
print("sampled action:", sampled_action, "log_prob:", round(math.log(sampled_prob), 3))

The worked example is small, but it shows the contract. The policy must not only choose an action; it must remember how surprising that action was under the current parameters. Without that record, the optimizer cannot say whether the action should become more or less likely after the return is known.

Practical Recipe

1. Choose the policy distribution to match the actuator: categorical for discrete actions, Gaussian or squashed Gaussian for continuous commands.
2. Log observations, sampled actions, rewards, terminations, and old log probabilities for every rollout step.
3. Keep exploration physically plausible by bounding actions and standard deviations before the command reaches the controller.
4. Evaluate with the same initial states and perturbations whenever two policy updates are compared.
5. Separate policy failure from execution failure by logging controller saturation, contact slips, timeouts, and safety stops.

Naming those fields and failure modes, as the self-check asks, is really a question about how the distribution itself is shaped, because every entry in the rollout buffer is a consequence of that choice. A direct policy optimizer only sees the consequences of actions that were actually sampled. That makes distribution design a systems decision, not a cosmetic modeling choice. Too little entropy prevents discovery; too much entropy spends rollouts on unsafe or uninformative behavior.

For embodied agents, the action distribution also sits between learning and control. A Gaussian policy over joint velocity commands may sample a value that the safety layer clips. If training records the unclipped log probability but the robot executes the clipped command, the update credits the wrong action. The implementation must log both the policy sample and the executed command. The table below summarizes how the policy form and its embodied cautions change with the action space.

Before reading on, predict: if your policy clips a sampled joint-velocity command from 1.35 rad/s down to 1.00 rad/s and stores the log probability of 1.35, what happens to the gradient signal? Check your answer against Code Fragment 2 below.

A robust implementation treats action sampling as part of the evidence artifact. Code Fragment 2 sketches the fields a rollout buffer needs before PPO or REINFORCE can make a valid policy-gradient update.

1. Store the observation before action sampling, not a later state estimate after the controller has moved.
2. Store the sampled action, executed action, old log probability, reward, value estimate, and termination flag in one row.
3. Record the random seed and policy version that produced the rollout.
4. Reject rollout rows where safety clipping or controller failure makes the training target ambiguous, or mark them with an explicit failure label.
5. Compare policies only when rollouts use the same reset distribution and perturbation suite.

# Define the rollout row that direct policy optimization needs.
# Store both sampled and executed actions so safety clipping is visible.
from dataclasses import dataclass, asdict

@dataclass
class RolloutRow:
    observation_id: str
    sampled_action: float
    executed_action: float
    old_log_prob: float
    reward: float
    clipped_by_safety: bool

    def as_row(self) -> dict[str, object]:
        return asdict(self)

row = RolloutRow(
    observation_id="episode_0042_step_0017",
    sampled_action=1.35,
    executed_action=1.00,
    old_log_prob=-0.42,
    reward=0.8,
    clipped_by_safety=True,
)
print(row.as_row())

When direct policy optimization fails, first inspect the action distribution before blaming the optimizer. Check entropy, action clipping, invalid-action masking, controller saturation, and whether high-return episodes came from meaningful exploration or lucky resets. Then rerun a small perturbation panel with fixed seeds so the policy change is compared against the same embodied conditions.