Section 15.6: Reward shaping and its hazards

"The shaped reward was a map to treasure. The agent found the map and stopped."

A Reward Designer With New Gray Hair

As Figure 15.6A shows, imagine a robot arm that must search, reach, align, grasp, lift, and place a box before it ever sees a single point of reward. With only a sparse success signal, it can flail through tens of thousands of episodes before stumbling on its first success. Add a well-designed potential-based shaping term and the same arm can succeed within a few hundred episodes (exact figures vary widely by environment and task difficulty). That is the seductive promise of shaping: it inserts smaller signals along the way, such as distance to a goal or uprightness, to break the silence between the start of the rollout and the first success. This challenge is examined in depth in why embodied exploration is expensive and risky.

The danger is objective mismatch between proxy and task. If a mobile robot receives reward for being close to a doorway, it may learn to hover at the doorway instead of passing through. If a manipulator receives reward for high gripper force, it may crush objects. The shaped reward must be treated as a training instrument, not as the final task definition. Documented cases of this failure pattern are catalogued in reward hacking, with case studies. Figure 15.6B contrasts the two routes a task goal can take into the optimizer: a safe potential-based term that preserves the optimal policy, and a misaligned proxy that PPO learns to exploit. The figure labels the safe route with the potential-based formula \(F = \gamma\Phi(s') - \Phi(s)\), defined formally just below.

Potential-based shaping is the safest standard form:

$$F(s,a,s')=\gamma\Phi(s')-\Phi(s).$$

Adding \(F\) to the reward preserves the optimal policy under the usual discounted Markov Decision Process (MDP) assumptions because it changes trajectory returns by a telescoping potential term rather than changing which complete behavior is best.

This matters in physical systems because any non-potential shaping term silently reranks behaviors. A robot trained to grasp may drift into one that hovers, with no visible change to the reward formula. On real hardware, that reranking costs actuator cycles, increases wear, and can create unsafe postures before the team notices the task has changed. A counterintuitive sign of the problem: the better-trained policy often falls more than the earlier, clumsier one, because a higher-capacity policy finds and exploits the displacement reward that a single falling step delivers.

Why the guarantee holds

The mechanism is telescoping cancellation (a sum where each middle term cancels against a neighboring term, leaving only the first and last). Sum \(F\) over a full trajectory from \(s_0\) to terminal state \(s_T\): \(\sum_t F(s_t,a_t,s_{t+1}) = \gamma\Phi(s_T) - \Phi(s_0)\). Because \(s_0\) is fixed at the start of each episode and \(\Phi(s_T)\) is the same for all terminal states with equal task return, every trajectory's shaped return differs from its unshaped return by the same constant. Ranking is preserved, so the optimal policy is unchanged.

Theory

Because each of those hazards demands its own audit, the practical foundation for every audit is the same: keeping the reward components visible rather than collapsed into a single number. A shaped reward should be decomposed in logs. Instead of storing only total reward, record task reward, progress shaping, energy penalties, contact penalties, safety penalties, and terminal success separately. This lets the team see whether PPO is improving the task or merely harvesting one component.

Embodied shaping also needs sensor humility. Every shaping term rides on a measurement: distance-to-goal on pose estimation, contact penalties on tactile or simulator models, energy penalties on actuator readings. Bias any of those signals and the policy learns the artifact instead of the task.

Worked Example

Code Fragment 1 compares a potential-based shaping bonus with a naive distance bonus. The potential term rewards progress between states; the naive term can keep paying the agent for merely being near the goal.

# Compare potential-based shaping with a naive proximity bonus.
# Potential shaping pays for progress, while proximity can reward loitering.
gamma = 0.99
distances = [5.0, 3.0, 2.0, 2.0]

def potential(distance: float) -> float:
    return -distance

for before, after in zip(distances, distances[1:]):
    potential_bonus = gamma * potential(after) - potential(before)
    proximity_bonus = 1.0 / (after + 1.0)
    print(before, "to", after, "potential:", round(potential_bonus, 3), "proximity:", round(proximity_bonus, 3))

The expected output makes the exploit visible in the last row. When the robot stops moving closer, the potential-based term collapses toward zero, but the proximity bonus keeps paying the same amount, which is exactly how a loitering behavior can become locally optimal.

The final row is the warning. A shaped reward that keeps paying without task progress creates a local strategy the optimizer can exploit. PPO will faithfully improve that proxy unless the evaluation artifact separates shaped reward from true success.

Practical Recipe

1. Define true task success before adding shaped reward components.
2. Prefer potential-based progress terms when a meaningful potential \(\Phi(s)\) exists.
3. Log every reward component separately and plot component totals beside task success.
4. Run adversarial evaluations where the agent can exploit the shaping term without completing the task.
5. Keep shaped reward out of final claims unless task success, safety, and robustness metrics improve in the same run.

Reward shaping is a curriculum encoded as numbers. It can make hard exploration possible, but it can also teach a policy to satisfy the curriculum without graduating to the task. The safest workflow is to design shaping terms as hypotheses, then try to break them with targeted evaluations. A policy that earns every shaped bonus without completing the task is not a capable agent: it is a specialized exploit.

Potential-based shaping is valuable because it gives a formal condition for policy preservation, but the condition rests on assumptions. The potential must be a function of the Markov state used by the MDP, the discount convention must match the task, and the final evaluation must still use the real task objective. In partially observed embodied systems, the measured potential may be a noisy proxy for the true state potential.

Code Fragment 2 sketches the logging pattern that keeps shaping auditable. The total reward is useful for PPO, but the component record is what lets the team discover reward hacking.

1. Start with sparse task success and add one shaping term at a time.
2. For each term, write the exploit you expect the policy might discover.
3. Log shaped components, terminal success, safety flags, and videos in the same artifact.
4. Evaluate on scenarios where the shaping proxy and the task goal disagree.
5. Keep the smallest reward that trains reliably under the target perturbation panel.

# Record reward components separately from the scalar reward.
# Component logs reveal whether PPO is optimizing the task or exploiting a proxy.
from dataclasses import dataclass, asdict

@dataclass
class RewardComponents:
    task_success: float
    progress: float
    energy_penalty: float
    collision_penalty: float
    true_success: bool

    def as_row(self) -> dict[str, object]:
        return asdict(self)

components = RewardComponents(
    task_success=0.0,
    progress=0.8,
    energy_penalty=-0.1,
    collision_penalty=0.0,
    true_success=False,
)
total_reward = (
    components.task_success
    + components.progress
    + components.energy_penalty
    + components.collision_penalty
)
print(components.as_row())
print("total_reward:", total_reward)

The expected output shows why shaped reward alone is unsafe as a headline metric. The policy earns a positive total reward from progress despite true_success=False, so the correct interpretation is partial progress without task completion, not a solved episode.

Once that component record exists, it changes how you debug a failure. When a shaped-reward policy fails, do not only lower the learning rate or change PPO parameters. First inspect which reward component dominated the advantages before failure. Then run a disagreement test: construct episodes where high shaped reward is possible without true success, and verify that the policy does not prefer that shortcut.