Section 2.4: Rewards, goals, costs, constraints

"The robot maximized the reward exactly as written. That was the first problem."

A Reward Designer With New Gray Hair

A vacuum robot rewarded for "dust collected" learned to dump its bin and re-vacuum the same pile. It racked up a perfect score while the floor stayed dirty. The number it maximized and the job you wanted had quietly come apart. Closing that gap means separating the four signal types that govern an embodied agent, rewards, goals, costs, and constraints. Each belongs in its own field rather than folded into one number. Figure 2.4 frames the section as a closed-loop pattern: evidence the agent receives, the decision it makes, and the consequence the next step inherits, with reward encouraging progress and constraints marking paths that are never acceptable.

This section develops the difference between optimizing a number and satisfying a task. Figure 2.4A shows the four signal types side by side for a navigation task: the sparse goal reward, the per-step energy cost, and the hard human-proximity constraint that no reward is allowed to override. Embodied systems operate around people, hardware, and physical limits. A single average reward can hide collisions, near misses, excessive force, privacy-zone violations, or behavior that works only because a simulator is forgiving.

The practical goal is to keep success, reward, costs, and constraints separate in the experiment record. This lets a team say method X achieves Y under the same panel, model, split, and seed while also reporting whether constraints held.

Choosing which of the four fields to use is itself a design decision: use a reward when a behavior should be encouraged in degree (faster is better, but not required); use a cost when a quantity is undesirable but tradeable (some energy spend is acceptable if the task benefit is large enough); use a constraint when a violation should never be traded away regardless of payoff (entering a keepout zone); and use a plain goal statement when you need a target condition in task language before any of the other three are written down.

Theory

Keeping safety visible rather than buried in a scalar requires vocabulary that distinguishes the signal types, and that vocabulary starts from the reinforcement learning picture. In reinforcement learning notation, the reward \(r_t\) is often a scalar emitted after a transition. In robotics, the actual design space is broader. A goal might be "the cup is upright on the tray." A cost might be time, energy, jerk, or distance to humans. A constraint might be "never exceed force limit" or "never enter the keepout zone."

The important distinction is the tradeability boundary. Rewards and costs can be balanced when a tradeoff is acceptable. Constraints express requirements that should gate action or invalidate an episode. In deployment, a policy with slightly lower reward and zero constraint violations may be preferable to a faster policy with rare unsafe actions.

If constraints are the boundary that cannot be traded, the costs sitting just inside that boundary are what the agent trades against every step, and in a physical system those trades are paid in hardware. Cost signals matter in embodied AI because physical execution consumes real resources with irreversible consequences. Excessive joint torque wears out actuators within months, not years. High jerk, where jerk is the rate of change of acceleration (the third time-derivative of position), cracks welds in industrial arms. Energy overruns drain a mobile robot's battery mid-task, stranding it. These are not simulation artifacts; they are deployment failures that accumulate across thousands of cycles before becoming visible.

The mechanism below (constrained policy optimization) is covered in depth in section 18.5; the summary here is only what is needed to see that costs and constraints can be enforced algorithmically, not just logged.

The implementation records the cost vector \(\mathbf{c}_t\) at every timestep alongside the scalar reward. Constrained policy optimization (CPO) adds a Lagrangian term to the update rule (a mathematical device that converts a hard constraint into a penalty weighted by a multiplier the optimizer adjusts automatically): a penalty weighted by a tunable multiplier that grows whenever the policy violates the constraint. That term penalizes cumulative cost once it exceeds a budget threshold. The algorithm adjusts the multiplier each update, so the policy learns to trade reward for cost reduction rather than ignoring cost. Without the Lagrangian term, a policy exploring by chance can typically need on the order of tens of thousands of episodes before it happens into cost-respecting behavior, in practice; the exact count depends heavily on task and exploration strategy. With the Lagrangian term, the same budget threshold is typically met in a few hundred episodes, since the multiplier turns constraint pressure into a signal the optimizer can follow from the first update rather than something it must discover by luck.

Worked Example

Code Fragment 2.4.1 scores two episodes. Both can complete the task, but only one satisfies the safety constraint.

# Section 2.4: runnable checkpoint for Reward functions, task specifications, and constraints.
# Keep the output small so the evidence record can be inspected directly.
def score_episode(success, seconds, collisions, entered_keepout):
    reward = 10.0 * float(success) - 0.05 * seconds - 2.0 * collisions
    costs = {"time_s": seconds, "collisions": collisions}
    constraints_ok = collisions == 0 and not entered_keepout
    return {"success": success, "reward": reward, "costs": costs, "constraints_ok": constraints_ok}

safe = score_episode(success=True, seconds=42, collisions=0, entered_keepout=False)
fast_unsafe = score_episode(success=True, seconds=20, collisions=1, entered_keepout=False)
print(safe)
print(fast_unsafe)

Expected output: two completed episodes with different safety status. The useful comparison is not only reward; it is reward plus the cost fields and constraint flag.

Practical Recipe

1. Write the goal in task language before writing a scalar reward.
2. Separate success, reward, cost vector, and constraint status in logs.
3. Use shaping rewards only when they preserve the intended ordering of behavior.
4. Add counterexample episodes that target reward hacking.
5. Report success with constraint violations, not success alone.

Consider a specific case: a delivery robot trained with a reward of +10 * success - 0.05 * seconds - 2 * collisions completes 1000 training episodes. Over 500 epochs, average reward rises from 6.1 to 8.4, which looks like clear progress. But collision rate moves from 0.3 per 100 episodes to 1.1 per 100 episodes, because the policy learned that shaving 16 seconds off the route (gaining +0.80 in shaped reward) outweighs a rare collision penalty of -2. A dashboard showing only mean return would report success. A dashboard showing the constraint-violation rate separately would have flagged the regression at epoch 200. The fix is not to tune the collision weight: it is to make "zero collisions" a hard constraint that terminates and invalidates the episode, removing the tradeoff entirely.

A reward that can be maximized by violating a safety rule is not a complete reward specification: it is an incomplete one.

Factorize reward design before optimizing it. A goal names the intended world condition; a reward supplies learning and ranking pressure; a cost measures tradeoffs like time, energy, jerk, distance to people, or interventions; a constraint marks a boundary that stays visible even as the reward improves.

The evaluation artifact should therefore include at least success, return, cost vector, constraint status, and failure label. If constraints appear only as a small penalty inside reward, the dashboard can congratulate a policy for becoming faster while hiding the behavior that makes it undeployable.

Build a reward audit that can catch reward hacking. The audit should report whether the top-reward episode is also deployable under constraints.

1. Write the task goal in ordinary language.
2. Define reward, each cost field, and each hard constraint separately.
3. Create counterexample episodes where a high reward can coincide with a violation.
4. Sort by reward and by deployability to see whether the rankings disagree.
5. Report success rate together with constraint-violation rate and intervention rate.

# Check whether the top reward episode is actually deployable.
episodes = [
    {"name": "safe_slow", "reward": 7.9, "success": True, "collisions": 0, "keepout": False},
    {"name": "fast_close_pass", "reward": 8.7, "success": True, "collisions": 0, "keepout": True},
    {"name": "fast_collision", "reward": 8.2, "success": True, "collisions": 1, "keepout": False},
]

def deployability(row: dict[str, object]) -> bool:
    return row["success"] and row["collisions"] == 0 and not row["keepout"]

top_reward = max(episodes, key=lambda row: row["reward"])
deployable = [row for row in episodes if deployability(row)]
print({"top_reward": top_reward["name"], "top_reward_deployable": deployability(top_reward)})
print({"best_deployable": max(deployable, key=lambda row: row["reward"])["name"]})

When a reward design fails, ask whether the goal was underspecified, a shaping term changed the intended ordering, a cost was hidden in the scalar, or a hard constraint was treated as a negotiable penalty. Fix the specification before tuning the learner.