Section 10.3: Reward design and termination

"The agent optimized what you wrote, not what you meant. The termination condition decided how long it had to find the loophole."

A Reward Designer, Revising Again

What This Section Builds

Reward design and episode endings become operational when the learning signal and the stopping reason are both explicit. A reward tells the learner what behavior is being reinforced, while terminated and truncated tell the learner why the episode stopped.

The goal is to stop treating all endings as equal. Reaching the goal, dropping the object, violating a safety boundary, and hitting a time limit require different labels in both the learning loop and the experiment report.

Theory

A reward should be tied to the task construct, not to whatever sensor is easiest to measure. In a reach task, distance-to-target may help shape learning, but the success condition should still state the task boundary: target reached within tolerance, object stable, safety constraints respected.

What happens when your sparse reward fires once every 50,000 episodes and the robot arm never discovers success at all? Figure 10.3A sketches the answer for a door-opening task: a sparse terminal reward fires only on success, a dense potential-based shaping term supplies gradient throughout, and an early-termination condition cuts failed episodes short.

Gymnasium separates two ending flags because they answer different questions. terminated means the task reached its own terminal state. truncated means an outside condition, usually a time limit, stopped the episode before the task ended. The distinction shapes value learning. The TD target (temporal-difference target) is the value the critic is trained to match at each step: the immediate reward plus, unless the episode has truly ended, a discounted estimate of the value remaining. At a terminated state, future return is zero by definition, so the TD target is just the immediate reward. At a truncated state the task is still ongoing, so the correct TD target adds a bootstrapped value estimate for the final observation, that is, a prediction from the critic network of how much reward remains, used as a stand-in for the return the agent would have collected had the episode continued. Algorithms that ignore this distinction systematically underestimate value for tasks with generous time limits. The resulting policies succeed within a few hundred steps but fail to exploit longer episodes even when given them. A policy trained with merged flags typically performs worse when you give it more time: its critic learned to treat every episode boundary as a dead end, so it stops planning past that boundary. A dense potential-based shaping term compresses sample cost dramatically. A sparse-only reach task may require 50,000 episodes before the agent finds the goal even once, while a distance-shaping term typically cuts that to under 500 episodes, a 100x reduction before any hyperparameter tuning (figures typical of benchmark reach tasks as of 2024; exact numbers vary by environment and robot morphology).

That hundredfold reduction is a software convenience in a simulator, but on hardware the same number sets the physical cost of training. For embodied agents, sample efficiency is not abstract. Every failed episode wears the joints, grippers, and cables, and a sparse reward can demand tens of thousands of failures before the arm ever finds the goal. Dense shaping supplies a learning gradient from the first step, so the policy improves each episode instead of random-walking into the goal region. Cutting required episodes from 50,000 to 500 is the difference between a viable training run and a destroyed actuator.

The potential-based shaping mechanism works by computing the difference in a scalar potential function between consecutive states: \(r_{\text{shape}} = \gamma \Phi(s_t) - \Phi(s_{t-1})\). When the potential \(\Phi\) is chosen as negative distance to goal, this difference is positive whenever the agent moves closer and negative when it moves away, producing a moment-to-moment progress signal. The \(\gamma\) discount factor in the formula is critical: it ensures the cumulative shaped reward over any trajectory changes the value function by a constant offset, leaving the ordering of policies unchanged. Only potential-based shaping carries this optimality-preservation guarantee; arbitrary dense rewards can redirect the agent toward a proxy rather than the true goal.

Worked Example

Shaping decides how fast the agent learns, but the termination flags decide how each episode is labeled once it ends, so the next example turns from reward construction to making that labeling concrete.

Code Fragment 10.3.1 forces a time-limit truncation in CartPole-v1. The pole has not necessarily reached a terminal failure state, but the wrapper stops the episode because the external step budget is exhausted.

# Show that a time limit produces truncation, not task termination.
# Learning code should log which flag caused the episode boundary.
import gymnasium as gym

env = gym.make("CartPole-v1", max_episode_steps=3)
observation, info = env.reset(seed=5)
env.action_space.seed(5)

for step_index in range(5):
    action = env.action_space.sample()
    observation, reward, terminated, truncated, info = env.step(action)
    print(step_index + 1, terminated, truncated)
    if terminated or truncated:
        break

env.close()

The expected output keeps both ending flags false for the first two steps, then flips only truncated to true at step 3. That pattern means the episode stopped because the time budget expired, not because the task dynamics reached a terminal success or failure state.

Practical Recipe

1. Write the terminal success and terminal failure conditions before writing reward shaping.
2. Use terminated for task-defined endings such as success, irreversible failure, or safety violation.
3. Use truncated for external cutoffs such as time limits, evaluation budgets, or watchdog stops.
4. Log reward components in info during debugging, even if the trainer only consumes the scalar reward.
5. Report success rate, truncation rate, and failure category counts together.

Reward design is where a simulator becomes a teacher. If the scalar reward rewards the wrong proxy, the policy will optimize that proxy with more patience than the author expects. The environment should therefore store the reward decomposition and ending cause with each episode. A reward function that is easy to measure but hard to verify is not a specification: it is an invitation for the optimizer to outpace the designer.

The graduate-level habit is to treat reward as a measurement model. The scalar is only a compressed signal. The full evidence artifact should retain the terms that explain the compression, especially when comparing methods across seeds or perturbations.

A robust implementation keeps reward computation auditable. When a Stable-Baselines3 PPO policy trained on an Isaac Lab Franka reach task plateaus, the scalar reward alone cannot tell you whether the agent is parking near the goal to harvest distance-shaping bonuses, racking up control-cost penalties from jittery joint torques, or repeatedly tripping the wrist force-torque safety predicate. The environment should emit each of those terms into info so the failure mode is reconstructible from the logged episode rather than rediscovered by re-running on hardware and risking another deformed gripper.

1. Write success and failure predicates before tuning reward weights.
2. Put every reward term into a named variable before summing.
3. Return the scalar reward to the trainer and the term breakdown in info.
4. Write unit tests for one success case, one task failure, and one time-limit truncation.
5. Audit evaluation tables for construct-matched success metrics, not only mean reward.

# Keep reward terms named before summing them into one scalar.
# The info dict preserves why the reward was assigned.
distance_to_goal = 0.04
collision = False
success = distance_to_goal <= 0.05 and not collision

reward_terms = {
    "success_bonus": 10.0 if success else 0.0,
    "distance_penalty": -distance_to_goal,
    "collision_penalty": -5.0 if collision else 0.0,
}
reward = sum(reward_terms.values())
terminated = success or collision
truncated = False
info = {"reward_terms": reward_terms, "success": success}

print(round(reward, 2), terminated, truncated)
print(info)

The expected output pairs a near-10 reward with terminated=True and a reward-term dictionary whose entries sum to the scalar reward. Readers should interpret that as a successful terminal transition whose score is explainable from named components rather than an opaque single number.

When an experiment about reward design and termination fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.