Section 14.1: Learning from interaction; return and discounting

"A reward is an event; a return is a sentence. Discounting decides how much the ending shapes the meaning of the opening."

A Robot Balancing Its Future

This section links back to Chapter 7: Control for AI Practitioners and Chapter 10: Environments with Gymnasium and PettingZoo, then prepares the policy-gradient work in Chapter 15: Policy Gradient Methods and PPO. The goal here is not to memorize notation; it is to know exactly what experience tuple an embodied agent records and how delayed rewards become a single training signal.

A robot's grasp succeeds or fails, but the action that decided the outcome happened a full second earlier, buried in a stream of motor commands that all looked fine at the time. To learn from that, you need a single number that reaches backward through the episode and pins the right amount of credit on each command: that number is the return, and building it correctly is what this section is about. First we define the Markov Decision Process (MDP) and POMDP objects, then we derive the return used by value functions, then we test the arithmetic on a short robot episode.

The key question is practical: when a robot receives a reward several seconds after a motor command, how should the learning system assign that reward to earlier actions without pretending the future is as certain as the present?

Theory

An MDP is the cleanest mathematical starting point for interaction. It is a tuple \((\mathcal S,\mathcal A,P,R,\gamma)\): states \(s \in \mathcal S\), actions \(a \in \mathcal A\), transition probabilities \(P(s' \mid s,a)\), rewards \(R(s,a,s')\), and a discount factor \(\gamma \in [0,1)\). The Markov assumption says the current state contains all task-relevant history: once \(s_t\) and \(a_t\) are known, older states do not change the distribution of \(s_{t+1}\).

For embodied agents, this assumption is not free. A robot arm whose state omits joint velocities violates the Markov assumption because the next position depends on current velocity, not just current position. When the assumption breaks, a policy trained on an incomplete state representation will receive different next states for identical actions and will never converge to consistent behavior. The design consequence is physical: every sensor reading whose omission would make the next state unpredictable must be included in \(s_t\), or the MDP formalism loses its guarantee.

Mechanically, the Markov assumption requires \(s_t\) to be a sufficient statistic of the trajectory so far. The transition kernel \(P(s' \mid s, a)\) depends only on the pair \((s_t, a_t)\) because \(s_t\) absorbs all causally relevant past information. In practice the state vector must encode velocity, contact status, and any latent quantity the next timestep's physics depends on. That way the mapping from \((s_t, a_t)\) to \(s_{t+1}\) becomes a well-defined probability distribution rather than a context-dependent lookup.

From Full State to Partial Observation

Embodied agents rarely receive the true state directly. A POMDP adds observations \((\mathcal O, O)\), where \(O(o \mid s)\) describes how sensors produce observations from hidden physical state. The robot may see pixels, joint encoders, force readings, and latency-corrupted messages, while the state also includes unobserved friction, object mass, and contact geometry. In practice, the policy acts on \(o_t\) or a belief/state estimate \(\hat s_t\), while the formal MDP remains the reference model for what the environment is doing.

Before reading on, guess: out of the inputs a warehouse robot receives (camera frames, joint encoder readings, force-torque sensor data, payload mass, floor friction), which ones are observations and which ones are hidden state? Most practitioners get at least one wrong on the first attempt.

Consider a concrete case: Boston Dynamics' Spot navigating a warehouse uses RGB-D cameras and joint encoders as observations, while the true state also includes floor friction, payload mass, and the exact positions of surrounding workers. The policy receives \(o_t\) (sensor readings) and must act without ever observing the full physical state \(s_t\). Google DeepMind's RGB-stacking experiments (Lee et al., 2021) face the same gap: the robot observes pixels, but the state includes object contact geometry and internal friction that no camera can fully resolve. Naming which inputs are observations versus latent state is one of the first practical decisions when applying this framework to a real robot.

A trajectory is the ordered interaction record \(\tau=(s_0,a_0,r_1,s_1,a_1,r_2,\ldots)\). As Figure 14.1A illustrates, interaction becomes learnable only when these observations, actions, rewards, and their later consequences are recorded together as one episode. The return from time \(t\) is

$$G_t = \sum_{k=0}^{\infty}\gamma^k r_{t+k+1}.$$

Figure 14.1B shows this sum geometrically: each reward is scaled by an increasing power of \(\gamma\) before the four contributions collapse into the single scalar \(G_t\).

Each term answers a credit assignment question. \(r_{t+1}\) is the immediate consequence of the current action; \(\gamma r_{t+2}\) gives the next consequence slightly less weight; \(\gamma^2 r_{t+3}\) gives the following consequence less weight again. A small \(\gamma\) teaches short-horizon reflexes. A large \(\gamma\) makes the policy care about delayed outcomes such as recovering balance after a stumble or avoiding a collision three actions later. The difference is stark. In practice, a legged locomotion policy trained with \(\gamma=0.9\) at 50 Hz typically needs on the order of 50,000 episodes to learn to avoid a delayed fall penalty, because the signal barely reaches the earlier motor commands. Raising \(\gamma\) to 0.99 keeps the same penalty visible across 100 steps and, in comparable setups, tends to cut that figure to roughly 300 episodes: every action in the run-up to the fall now carries a non-negligible piece of blame. These specific counts vary with reward scale and task, but the qualitative gap between the two regimes is typically large and in the same direction.

Worked Example

Consider a mobile manipulator with four measured consequences after a grasp attempt: a small movement cost, a contact bonus, a delayed placement reward, and a final safety penalty. Code Fragment 1 computes the return backward so the reader can verify exactly how each future consequence reaches the current action.

# Compute discounted returns for one embodied episode.
# Backward accumulation makes delayed placement and safety effects visible.
rewards = [-0.2, 0.4, 2.0, -1.0]
gamma = 0.9

returns = []
running_return = 0.0
for reward in reversed(rewards):
    running_return = reward + gamma * running_return
    returns.append(round(running_return, 3))

returns.reverse()
for t, (reward, discounted_return) in enumerate(zip(rewards, returns)):
    print(f"t={t}: reward={reward:+.1f}, G_t={discounted_return:+.3f}")

The expected output should be read from bottom to top as a backward return construction. The key interpretation is that the early negative reward at t=0 still receives a positive return because later placement success outweighs it after discounting, while the terminal penalty remains fully visible at t=3.

The first action has a negative immediate reward but a positive return because it helped set up later success. This is the central difference between reinforcement learning and one-step supervision: the learning target for an action is not only what happened immediately after it, but what the episode eventually made possible. A policy that sees only immediate rewards is not learning from experience; it is reacting to the nearest echo.

Practical Recipe

1. Fix the control frequency and episode length before picking \(\gamma\). A Franka Panda arm running at 1 kHz control with a 10-second manipulation episode has 10,000 steps; \(\gamma=0.999\) keeps a terminal reward visible at \(0.999^{10000} \approx 4.5 \times 10^{-5}\), which is marginal. Drop to 100 Hz by aggregating control commands and the math becomes tractable.
2. Log \((o_t, a_t, r_{t+1}, o_{t+1}, \texttt{terminated}, \texttt{truncated})\) as a flat CSV from the first simulator run, not as nested dicts. Isaac Lab and MuJoCo both produce vectorized episode buffers; inspect one row per episode end before training, not after a failed convergence run.
3. Spot-check the backward return computation on one hardware trace. For a Boston Dynamics Spot stair-climbing episode, \(G_0\) should be positive when the robot reaches the top and negative when it falls; if the sign is wrong, the termination rule or reward sign is inverted.
4. Separate truncation from termination in the bootstrap, where a bootstrap means substituting a learned value estimate for the unknown remainder of the return instead of assuming it is zero. In IsaacGym/Isaac Lab, episodes time-out at a fixed step count; every such boundary should use a value-function bootstrap, not a zero target. Treating 1,000-step timeouts as failures is the single most common source of pessimistic value estimates in legged locomotion training.
5. Run at least one sim-to-real sanity check on the reward: replay a hardware bag file through the same reward function used in simulation and verify the numbers are in the same range. A reward calibrated for MuJoCo's frictionless defaults will misscale on a real floor with \(\mu \approx 0.6\).

The MDP formalism is a modeling claim, not a fact about the robot. When the true physical state is hidden, the implemented learning system either uses observations directly or builds a belief/state estimate. State which one your system uses, because \(V^\pi(s)\) and \(V^\pi(\hat s)\) are different claims.

Just as the choice between \(V^\pi(s)\) and \(V^\pi(\hat s)\) fixes what the return is computed over, the discount factor fixes how far into the future that return reaches. Discounting also has a physical interpretation. With \(\gamma=0.99\), a reward 100 steps away still carries \(0.99^{100} \approx 0.37\) of its original weight; with \(\gamma=0.5\), that same reward shrinks to \(0.5^{100} \approx 10^{-30}\), effectively invisible. The practical consequence is stark: at 10 Hz control, \(\gamma=0.99\) keeps a collision that happens 10 seconds later visible in the training target, while \(\gamma=0.9\) makes it vanish after roughly 1 second. This is the discount horizon as a design parameter, not a tuning knob, and choosing the wrong value is one of the fastest ways to train a policy that ignores delayed consequences. Choose it to match the task horizon and controller rate, not as an inherited default.

Start by fixing the episode schema: each row holds observation, action, reward, next observation, termination, truncation, and any safety event. With that schema stable, one return computation serves simulator traces, replay buffers, and hardware logs alike.

1. Define the MDP or POMDP fields before collecting data.
2. Write the reward in units a domain expert can inspect.
3. Choose \(\gamma\) from task duration and control frequency.
4. Compute returns from saved traces and spot-check at least one episode by hand.
5. Store raw rewards and returns, since later algorithms may use different horizons.

With that explicit schema and recipe in place, most debugging reduces to checking the same fields you just defined. When return learning fails, first inspect the episode boundary and reward timing. Many apparent algorithm failures are target-construction failures: rewards arrive one step late, terminal penalties are dropped, time-limit truncations are treated as true failures, or observations do not contain the state needed for the Markov assumption.