Section 14.2: Policies and value functions

"The actor does not need to know why it was wrong, only that it was. The critic keeps score so the actor can keep moving."

A Two-Headed Learner

This section links back to Chapter 7: Control for AI Practitioners and Chapter 10: Environments with Gymnasium and PettingZoo, then prepares the policy-gradient work in Chapter 15: Policy Gradient Methods and PPO. The central distinction holds throughout: a policy chooses actions, while a value function estimates the future those actions create. Figure 14.2B shows how both objects take the state as input and how the Bellman equation links them.

The technical contract for policies and value functions follows in three parts: stochastic and deterministic policies, the Bellman equations that connect state values and action values, and a small policy-evaluation example.

The key question is practical: when an embodied agent stands in a state with several possible actions, how does it separate "what I will do" from "how good this situation is if I keep doing it"?

Theory

A policy maps information to actions. In a fully observed Markov Decision Process (MDP), a stochastic policy is written \(\pi(a\mid s)\), the probability of choosing action \(a\) in state \(s\). In a partially observed robot task, the implemented policy often uses \(\pi(a\mid o)\) or \(\pi(a\mid \hat s)\), where \(\hat s\) is the state estimate produced by perception and filtering.

The state-value function for a fixed policy is the expected return from state \(s\):

$$V^\pi(s)=\mathbb E_\pi[G_t\mid S_t=s].$$

The action-value function asks the same question after forcing the first action:

$$Q^\pi(s,a)=\mathbb E_\pi[G_t\mid S_t=s,A_t=a].$$

These functions obey Bellman expectation equations because the return can be split into immediate reward plus discounted future value:

$$V^\pi(s)=\sum_a \pi(a\mid s)\sum_{s'}P(s'\mid s,a)\left[R(s,a,s')+\gamma V^\pi(s')\right].$$

The equation is recursive, not circular. It says the value of a state under policy \(\pi\) equals the policy-weighted average of one-step outcomes plus the discounted value of the next state. For embodied systems, the hidden assumption is strong: the state or belief must contain enough physical information for the transition model to be meaningful. A value function trained on an incomplete state is not a forecast; it is a confident guess about a world the agent cannot fully see.

That same demand for a complete state shapes how aggressively the recursion should weight the future, which is exactly what the discount factor controls. For a physical robot, the discount factor \(\gamma\) is not merely a convergence device: it encodes how much joint wear, battery drain, or task-deadline pressure should reduce the weight of rewards earned ten steps from now. Set \(\gamma\) too high and the robot will accept dangerous joint trajectories to reach a distant goal; set it too low and it will ignore recoverable failure modes entirely because they are more than a few steps ahead.

Mechanically, each Bellman backup multiplies the next state's value by \(\gamma\) before adding the immediate reward. After \(k\) steps that factor accumulates as \(\gamma^k\), so a reward \(k\) steps away contributes only \(\gamma^k\) of its face value to the current estimate. At \(\gamma = 0.99\) a reward 100 steps ahead still carries 37% of its face value. At \(\gamma = 0.9\) that same reward shrinks to 0.003%, so the agent goes effectively blind beyond about 30 steps. In practice, this is why long-horizon manipulation tasks typically need \(\gamma \geq 0.99\) to learn useful recovery behavior; the exact threshold still depends on episode length and reward sparsity. The recursive structure means the algorithm never sums an infinite horizon explicitly: one sweep through all states applies the discount automatically, and repeated sweeps propagate that shrinking weight backward through the state space until estimates converge.

The Bellman equation converges to the correct values under two conditions: the discount factor \(\gamma\) must be less than 1, and the Markov property must hold. The recursive structure buys efficiency. Averaging full rollouts, known as Monte Carlo estimation (sampling entire episodes and averaging the observed returns, rather than bootstrapping from one-step backups), needs roughly 10,000 episodes to push that sampling noise below 1% on a 50-step task with \(\gamma = 0.99\). Iterative Bellman backups on the same task converge in under 50 sweeps, each touching every state once. The Markov property requires the current state to capture all information relevant to future returns. In embodied systems, both conditions are routinely violated. A discount factor of 1 makes the backup sum diverge for continuing tasks. A robot state that omits gripper force, contact normal, or task progress breaks the Markov property, so the "state" is really a partial observation and the backup targets carry systematic error. When policy evaluation diverges or produces physically implausible values, check these two conditions before suspecting the learning algorithm.

Worked Example

Code Fragment 1 evaluates a fixed two-state policy for a charging robot. The state `low` means the battery is low, and the state `ready` means the robot can inspect objects; the policy recharges aggressively when low and mostly inspects when ready.

# Evaluate a fixed policy with Bellman expectation backups.
# The values forecast long-run return without changing the policy.
states = ["low", "ready"]
gamma = 0.8
policy = {
    "low": {"recharge": 0.9, "inspect": 0.1},
    "ready": {"recharge": 0.2, "inspect": 0.8},
}
transitions = {
    ("low", "recharge"): ("ready", 1.0),
    ("low", "inspect"): ("low", -2.0),
    ("ready", "recharge"): ("ready", 0.2),
    ("ready", "inspect"): ("low", 3.0),
}

values = {state: 0.0 for state in states}
for _ in range(8):
    new_values = {}
    for state in states:
        total = 0.0
        for action, prob in policy[state].items():
            next_state, reward = transitions[(state, action)]
            total += prob * (reward + gamma * values[next_state])
        new_values[state] = total
    values = new_values

for state in states:
    print(f"V({state}) = {values[state]:.3f}")

The numbers are not labels supplied by a dataset. They are self-consistent forecasts under the policy. If the policy changed, the Bellman equations would describe a different behavior and the values would need to be recomputed.

The example above computes \(V^\pi\), but \(Q^\pi(s,a)\) answers a narrower question: what is the forecast if the first action is fixed rather than drawn from the policy? Using the converged values \(V(\text{low}) = 6.161\) and \(V(\text{ready}) = 7.266\), the action-value of forcing recharge from low is \(Q^\pi(\text{low},\text{recharge}) = 1.0 + 0.8 \times 7.266 = 6.813\), and the action-value of forcing inspect from low is \(Q^\pi(\text{low},\text{inspect}) = -2.0 + 0.8 \times 6.161 = 2.929\). The policy's own weighting, \(0.9 \times 6.813 + 0.1 \times 2.929 = 6.425\), lands close to but not exactly at the printed \(V(\text{low}) = 6.161\) because the sweep count and rounding in Code Fragment 1 leave a small residual; the two objects are consistent by construction, and \(Q^\pi\) is what a policy-improvement step would compare across actions to decide whether recharge or inspect should be favored more heavily from low.

Practical Recipe

1. Define the observation vector explicitly before writing any training code: for a Franka Panda manipulation task this typically means 7 joint positions, 7 joint velocities, 6-axis end-effector wrench, and a 3D target pose (25 floats total). Missing the wrench signal is, in practice, one of the most common causes of value functions that look converged in simulation but collapse on hardware because they cannot detect imminent slip.
2. Start with a tabular or two-layer MLP (multi-layer perceptron, a small fully-connected neural network) critic in Isaac Lab or MuJoCo before adding a CNN or transformer. A 128x128-hidden MLP converges in under 10 million steps on FrankaKitchen; a premature visual backbone adds 40+ million steps and hides state-representation bugs behind capacity.
3. Confirm the Gymnasium terminated vs. truncated split is wired correctly: set bootstrap weight to zero only on terminated. On a 200-step horizon task (typical for MuJoCo Ant or HalfCheetah), conflating the two flags depresses value estimates by roughly \(\gamma^{200} \approx 0.13\) for \(\gamma = 0.99\), which is large enough to reverse the advantage sign for long recoveries.
4. Record failures tagged by physical category: perception error (wrong object pose from depth noise), state error (missing contact normal), planning error (value-driven choice that ignores kinematic limits), control error (PD gains too low for commanded torque), or evaluation error (sparse reward never fires). This taxonomy maps directly to the three objects, \(\pi\), \(V^\pi\), \(Q^\pi\), to speed up ablations.
5. Run at least one sim-to-real perturbation before trusting the value function: add 5 ms observation latency, 10% joint-angle noise, and randomized friction (0.5 to 1.5x nominal) in Isaac Lab's domain randomization API. A value function that changes by more than 20% under these perturbations will produce unsafe action rankings on a physical Spot or UR10 before the first hardware episode.

Why the machinery earns its place

With the policy and value objects now clearly separated, it is worth stepping back to why the machinery that connects them earns its place. Bellman equations are useful because they turn a long-horizon prediction into repeated one-step backups. The backup is only as good as the state representation it conditions on. If a value function sees a camera frame but not the gripper force that predicts slip, it may assign the same value to physically different situations.

For policy-gradient methods in the next chapter, the value function often becomes a baseline or critic. That critic is not an optional decoration: it controls variance, estimates advantage (how much better an action is than the state's average value, \(A^\pi(s,a) = Q^\pi(s,a) - V^\pi(s)\)), and can quietly mislead the update when trained on the wrong rollout distribution.

Keep policy outputs, value targets, and evaluation returns in separate logs. Otherwise a good action gets read as proof the value estimate was right, and the two failures hide each other.

1. Define whether the policy receives \(s\), \(o\), or \(\hat s\).
2. Log action probabilities or deterministic actions for each decision.
3. Compute Monte Carlo returns from traces before fitting a value network.
4. Compare Bellman targets and observed returns on a small batch.
5. Audit high-value states for physical plausibility with video or state logs.

When a value function fails, compare three quantities for the same episodes: predicted value at the start, realized discounted return, and the first few Bellman targets. Large disagreement can indicate reward bugs, truncation bugs, distribution shift, or missing state information.