Section 14.5: Why RL is hard in embodied systems (sample cost, reward, safety)

"Bandit benchmarks count regret in milliseconds. Embodied RL counts it in broken joints, sparse rewards, and the afternoon you spent re-homing a policy that learned to cheat."

A Benchmarker Who Bought Hardware

The control tools in Chapter 7: Control for AI Practitioners and the environment interface in Chapter 10: Environments with Gymnasium and PettingZoo set up the policy-gradient work in Chapter 15: Policy Gradient Methods and PPO. The formalism from the earlier sections becomes difficult once samples are physical, rewards are imperfect, and safety is nonnegotiable.

Picture a policy that scored flawlessly in simulation crushing a $40 silicone fingertip on its very first real grasp, then demanding a human technician to re-home the arm before it can try again: that single moment is the whole problem of embodied RL, where the three obstacles of sample cost, reward design, and safety constraints collide. This section develops the technical contract behind that moment. First we connect these issues to the MDP and POMDP assumptions, then we express safety as a constrained objective, then we run a small numeric audit.

The key question is practical: what makes a high-return policy unacceptable when the learning process itself can damage hardware, surprise nearby people, or exploit a reward proxy?

A policy that earns high reward in simulation but destroys a gripper on the first real contact is not a solution; it is a simulation artifact dressed up as one. Figure 14.5A captures this tension: embodied RL has to earn reward without hiding sample cost, reset burden, or safety violations. Figure 14.5B then breaks that tension into the three concrete obstacles this section treats in turn.

Theory

Sample cost is the first obstacle. In a simulator, a failed episode may cost milliseconds. On hardware, it may cost a reset, a worn gripper, a human intervention, or a damaged object. A typical manipulation task might need 50,000 simulator episodes. On real hardware that same count would take months of continuous operation and wear through multiple grippers. Demonstration-seeded initialization drops that figure to roughly 300 real episodes for the same task and the same policy class, because the agent starts near successful behavior instead of exploring from scratch. This shifts the acceptable exploration policy, the number of seeds, and the evaluation budget.

Reward design is the second obstacle. The reward \(R(s,a,s')\) is a proxy for the task, not the task itself. A robot rewarded for moving a block near a target may learn to shove it violently, exploit perception blind spots, or end in unstable poses that score well for one frame. This failure mode is called reward proxy collapse, and it is the expected outcome whenever maximizing the proxy is easier than completing the real task. Sparse rewards delay credit assignment; dense rewards can teach the wrong shortcut.

Why sparse reward bites harder on hardware

Sparse rewards matter in embodied AI because a physical robot cannot re-run a failed episode in milliseconds. When reward arrives only at task completion, a robot attempting to open a door may execute thousands of real joint motions across hours of hardware time without ever receiving a signal that distinguishes useful from useless behavior. Each wasted episode consumes battery, wear budget, and reset labor. The cost of credit-assignment failure is not just slow convergence; it is hardware degradation and exhausted sample budgets before learning has started.

The mechanism is temporal distance between action and outcome. When reward is zero for every step except the final success, gradient estimates become high-variance. The policy cannot distinguish which of the hundreds of preceding actions contributed to the result. Long zero-reward trajectories produce near-zero policy gradients, so parameter updates stay negligibly small. The agent explores without improving until a rare successful trajectory delivers signal. On physical hardware, that lucky episode may never arrive within the available budget.

Safety is the third obstacle. A constrained MDP, where the standard reward-maximizing MDP gains a second cost signal and a budget the policy may not exceed, writes the builder's intent more explicitly:

$$\max_\pi J_R(\pi)=\mathbb E_\pi\left[\sum_{t=0}^{\infty}\gamma^t r_{t+1}\right]\quad\text{subject to}\quad J_C(\pi)=\mathbb E_\pi\left[\sum_{t=0}^{\infty}\gamma^t c_{t+1}\right]\le d.$$

Here \(r\) is task reward, \(c\) is safety cost, and \(d\) is the allowed discounted cost budget. The constraint matters because a policy can have excellent reward and still be unacceptable if it reaches that reward through collisions, excessive force, or unstable contacts. A hard constraint like this cannot be plugged directly into a gradient-based optimizer, so in practice it is converted into a soft, differentiable penalty via a Lagrange multiplier \(\lambda\), a scalar weight that is itself learned during training: the algorithm subtracts \(\lambda\) times the cost from the reward objective, and \(\lambda\) grows whenever the cost exceeds the budget. This is the mechanism the tip and algorithm below refer to as "Lagrangian."

Worked Example

Code Fragment 1 evaluates three candidate policies with the same reward and safety-cost definitions. The best policy by reward is not automatically acceptable because the safety budget is a separate constraint.

# Audit reward and safety cost for three embodied RL policies.
# A policy passes only if return is high and discounted cost stays within budget.
gamma = 0.9
safety_budget = 0.45
episodes = {
    "careful": {"rewards": [0.2, 0.5, 1.0], "costs": [0.0, 0.0, 0.1]},
    "fast": {"rewards": [0.4, 1.2, 1.8], "costs": [0.0, 0.4, 0.6]},
    "reckless": {"rewards": [1.0, 1.0, 2.0], "costs": [0.3, 0.5, 0.8]},
}

def discounted_sum(values):
    return sum((gamma ** t) * value for t, value in enumerate(values))

for name, trace in episodes.items():
    reward_return = discounted_sum(trace["rewards"])
    safety_cost = discounted_sum(trace["costs"])
    status = "pass" if safety_cost <= safety_budget else "reject"
    print(f"{name}: return={reward_return:.2f}, cost={safety_cost:.2f}, {status}")

The expected output ranks the policies two ways at once: by return and by discounted safety cost. The correct interpretation is that careful is the deployable winner because it stays under budget, while the higher-return policies fail the constraint and therefore do not count as acceptable solutions.

The result is a wins-only lesson for deployment: report the policy that satisfies the safety constraint and achieves the strongest validated return. Keep unsafe exploratory candidates in the experiment registry and diagnostics, not in the headline result.

Practical Recipe

1. Before any on-hardware training, estimate your physical sample budget: for a Franka Panda gripper task, 50,000 on-robot episodes at 5 seconds each is roughly 70 hours of continuous operation and will wear through 2-3 silicone fingertip pads. Route any policy that needs more than ~5,000 hardware episodes to Isaac Sim or MuJoCo first, using domain randomization over friction (0.5-1.5x nominal) and payload (0.8-1.2x nominal) to bridge the sim-to-real gap.
2. Define your observation space around what the physical sensor can actually deliver: a Realsense D435 gives 30 Hz RGB-D at 640x480; a wrist-mounted FT sensor (e.g., ATI Mini45) gives 6-axis wrench at 1 kHz. Build the baseline policy around these concrete data rates, not an idealized state vector.
3. Add the constrained policy (PPO-Lagrangian or TRPO-Lagrangian via Safety Gymnasium) only after a scripted fallback controller confirms the task is reachable and the reset procedure is reliable.
4. Record failures as structured cases with physical cause: joint-limit breach, contact force exceeding the platform's rated limit (e.g., 87 N for Spot's foot), perception dropout from motion blur above 0.5 m/s end-effector speed, or human E-stop trigger.
5. Run at least one perturbation rollout with lighting changed by 50 lux and object pose perturbed by 3 cm before treating any result as transferable.

Specifying those five quantities is only the first half of the work; the formal objective that ties them together hides several engineering costs that the clean notation does not show. The expectation in \(J_R(\pi)\) assumes the agent can sample trajectories from the environment. In physical systems, each trajectory consumes calendar time, reset labor, battery cycles, hardware wear, and safety margin.

The reward and cost functions should be reviewed like interfaces. A reward that ignores force can favor damaging contact. A cost that triggers only after collision misses near-miss behavior. A reset procedure that changes object distribution can make evaluation easier than deployment.

A robust implementation treats safety and sample cost as first-class outputs. The experiment answers four questions on demand: how many physical trials it used, how often a human intervened, how many constraints it violated, and which reward components drove the final policy.

1. Define reward and safety cost separately before training.
2. Set an allowed safety budget and a stop rule for violations.
3. Log resets and interventions with timestamps and causes.
4. Audit reward components for proxy shortcuts on saved videos or traces.
5. Report only policies that satisfy the safety budget on the shared evaluation panel.

Those ledgers do more than document a run; they let you diagnose what went wrong, because each failure leaves its signature in a different ledger. When embodied RL fails, classify the failure before tuning. A sample-cost failure (the Franka exhausts its 5,000-episode hardware budget before convergence) calls for demonstration-seeded initialization or an Isaac Sim pretrain, not more hyperparameter sweeps. A reward failure (the Spot policy from Krakovna et al.'s specification-gaming catalogue learns to topple toward the goal rather than walk to it) calls for a repaired proxy or shaped force penalty. A safety failure (PPO-Lagrangian in Safety Gymnasium drifts past its cost_limit because the multiplier updated too slowly) calls for a control-barrier shield (a safety filter that overrides the policy's action whenever it would cross a predefined safe-state boundary) or a stricter early-stop callback, not a higher learning rate.