Section 18.2: Sparse vs. dense; shaping done right

"Sparse rewards define the destination. Dense rewards pave a road that sometimes leads somewhere else."

Section 18.2

Imagine telling a robot "you win only when the block is in the bin" and nothing else. It could flail for fifty thousand episodes before stumbling onto success even once. A single extra line of reward math can cut that to a few hundred, or quietly teach it to cheat forever. This section gives you that line of math, shows you how it goes wrong, and hands you the audit that catches the cheat: you will add dense feedback to a sparse task using potential-based shaping (a reward modification of the form \(\gamma\Phi(s')-\Phi(s)\) that provably preserves the optimal policy), audit a flawed shaped reward, and repair it so learning speeds up without redefining the task (the hands-on version of this audit is the FetchReach lab later in this section). Figure 18.2A contrasts the two regimes at a glance: a sparse signal that fires only at success versus a shaped dense signal that supplies a gradient toward the goal at every state.

This section develops the contract for shaping rewards without rewriting the task. Sparse rewards give credit only at success or failure. Dense rewards add intermediate feedback, such as distance-to-goal progress, uprightness, clearance, or energy use. The difference in practice is stark. A gripper learning a sparse pick-and-place task may need tens of thousands of episodes to first succeed by chance. The same task with a well-designed potential-based shaping term reaches equivalent success in a fraction of that budget; manipulation benchmarks typically report empirical speedups in the range of 10x to 30x, though the exact ratio depends heavily on task geometry and potential choice. Without shaping, a gripper may need 50,000 episodes just to stumble on success once. With a well-designed distance potential, the same first success arrives around episode 300, because every approach toward the object now returns a nonzero gradient rather than silence.

On a physical robot, this gap has direct hardware consequences. A gripper that needs 50,000 episodes to first succeed by chance accumulates 50,000 episodes of joint wear, motor heat, and potential collision events before the policy begins to improve. Simulation can absorb that cost; real steel cannot. Sparse rewards are not merely slow; they are mechanically expensive in a way that shapes which tasks are feasible to train on hardware at all. A reward signal that tells the robot nothing until success is not a teacher; it is a silent witness to every failed attempt. The diagram below makes the mechanism concrete: it plots reward against trajectory step for the sparse case (left panel) and the potential-based dense case (right panel), showing where the nonzero shaping term \(F(s,a,s')\) enters.

The gradient absence is the mechanism behind sparse difficulty. With a sparse reward, the policy gradient estimator sees zero return for every episode that does not reach success, so the gradient estimate is effectively noise averaged over mostly-zero returns. Dense shaping injects a nonzero return into every transition, giving the gradient estimator a direction to follow even when the agent is far from the goal. The potential-based form ensures that direction points toward the task rather than toward a convenient surrogate.

The key question is practical: does the extra feedback make learning easier while preserving the policy ranking implied by the real success condition?

Theory

Potential-based shaping is the standard way to add dense guidance while preserving optimal policies under the usual discounted Markov Decision Process (MDP) assumptions. Choose a scalar potential \(\Phi(s)\) that measures progress in a state. Then add the shaping term

$$F(s,a,s') = \gamma \Phi(s') - \Phi(s).$$

The shaped reward is \(r'(s,a,s') = r(s,a,s') + F(s,a,s')\). The intuition is the telescoping cancellation property: along a trajectory, the added terms mostly cancel, leaving a boundary term tied to the start and finish rather than a new preference for a particular path. This is why a distance-like potential can speed learning without paying the agent forever for pacing near the goal.

Worked Example

Suppose a gripper starts three grid cells from a target. The sparse reward is zero until the final success, but the potential \(\Phi(s)=-\text{distance}(s,\text{goal})\) produces a small positive shaping reward whenever distance shrinks. Code Fragment 1 shows the actual numbers.

# Compute potential-based shaping for a short reaching path.
# Progress creates dense feedback while final success remains separate.
gamma = 0.9
distances = [3, 2, 1, 0]
sparse_rewards = [0, 0, 1]

for step, reward in enumerate(sparse_rewards):
    phi_now = -distances[step]
    phi_next = -distances[step + 1]
    shaping = gamma * phi_next - phi_now
    shaped_reward = reward + shaping
    print(step, "sparse=", reward, "shaping=", round(shaping, 2), "shaped=", round(shaped_reward, 2))

Expected output: intermediate shaped rewards appear before success, but the unshaped success reward remains visible. That separation is what lets the evaluation report learning speed and task success without mixing them.

Keeping those two signals apart is exactly the discipline the following recipe formalizes into a repeatable sequence of design steps.

Practical Recipe

1. Start with the sparse success condition that matches the task.
2. Identify the learning bottleneck: exploration, credit assignment (the algorithm box below defines each of these three bottleneck categories precisely), or delayed feedback.
3. If adding a dense term, write whether it is potential-based or a deliberate task change.
4. Log raw reward, shaping term, and shaped reward separately.
5. Report final success using the unshaped metric, then report shaping as a training aid.

The shaped reward should be treated as a training interface, not as the headline metric. During optimization, the learner sees \(r'\). During evaluation, the report should expose raw task reward, shaping contribution, safety cost, and embodied success. This prevents a shaped run from looking better simply because it received more arithmetic along the way.

The graduate-level habit is to state the assumption behind the invariant. Potential-based shaping preserves optimal policies for the same discounted Markov decision process when the shaping term has the form \(\gamma\Phi(s')-\Phi(s)\). If the potential depends on hidden evaluator state, future information, changing curricula, or nonstationary human hints, the invariance argument no longer applies cleanly.

Honoring that invariant in practice depends less on theory than on the tooling you reach for, so it helps to know which libraries make the shaped-versus-unshaped separation easy to maintain.

A robust implementation keeps shaping mechanically separate from task success. The environment can compute both, the policy can train on the shaped reward, and the evaluator can report success without the shaping bonus.

1. Define the sparse success event and termination condition first.
2. Choose a potential with physical units, such as negative distance or negative height error.
3. Compute the shaping term from consecutive states, not from the action label alone.
4. Log a per-step reward decomposition table.
5. Compare shaped and unshaped runs on one seed panel and one success definition.

Code Fragment 2 records the fields that should appear beside any shaped-reward experiment.

# Build one shaping audit record for a robot reaching task.
# The invariant field states why the dense term should not change the goal.
from dataclasses import dataclass, asdict

@dataclass
class ShapingAudit:
    section: str
    sparse_success: str
    potential: str
    shaping_formula: str
    report_metrics: list[str]

    def as_row(self) -> dict[str, object]:
        return asdict(self)

record = ShapingAudit(
    section="18.2",
    sparse_success="object within 2 cm of target for 10 consecutive steps",
    potential="negative gripper-to-target distance in meters",
    shaping_formula="gamma * Phi(next_state) - Phi(current_state)",
    report_metrics=["raw_success_rate", "mean_shaping_return", "collision_cost"],
)
print(record.as_row())

When shaping fails, first check whether the policy optimized a dense term easier than success, then compare the best shaped rollouts against unshaped success, safety cost, and final state. Label the failure precisely: a bad potential, a nonstationary hint, a missing cost, or a metric report that hid the raw reward.