Section 1.3: Agents, environments, observations, actions, rewards, constraints

"Interfaces matter because the agent can only act on what the experiment lets it see and do."

A Careful Control Loop

The task contract is a tuple

Six words, written down before a single line of algorithm exists, decide whether a robot learns the task you meant or quietly optimizes a different one: agent, environment, observation, action, reward, constraint. Figure 1.3 sketches the whole arrangement as a closed loop before we formalize it: the agent receives evidence, makes a decision, and inherits the consequence as its next input. The agent is the decision-maker itself: the policy that maps what it has seen to what it does next; the environment is everything outside that policy, including the physics, the objects, and any other agents, that determines how the world responds. The base contract for a fully observed embodied task is a Markov decision process, the tuple \((\mathcal{S},\mathcal{A},P,r,\gamma,\rho_0)\). Here \(\mathcal{S}\) is the state space (every configuration the world can be in: joint angles, object poses, contact state), \(\mathcal{A}\) is the action space, \(P(s' \mid s,a)\) is the transition kernel giving the probability of landing in state \(s'\) after taking action \(a\) in state \(s\), \(r(s,a)\) is the scalar reward, \(\gamma \in [0,1)\) is the discount factor, and \(\rho_0\) is the initial-state distribution. A policy \(\pi(a \mid s)\) closes the loop. The Markov property is the load-bearing assumption: \(s_t\) summarizes all history relevant to the future, so \(P\) and \(r\) may condition on \(s_t\) alone.

The objective is the discounted return, and the policy is graded by its expected value:

$$G_t = \sum_{k=0}^{\infty} \gamma^{k} r_{t+k}, \qquad J(\pi) = \mathbb{E}_{s_0 \sim \rho_0,\, a_t \sim \pi,\, s_{t+1}\sim P}\!\left[\sum_{t=0}^{\infty} \gamma^{t} r(s_t,a_t)\right].$$

Two distinctions in this objective are routinely conflated and cost teams weeks. Reward \(r_t\) is a per-step quantity; return \(G_t\) is the discounted sum the policy actually optimizes. A reward that looks correct step by step can induce a return that rewards stalling, oscillating, or any behavior the discount happens to favor. And the expectation is taken over the policy-induced trajectory distribution, so changing \(\pi\) changes the states on which \(r\) is sampled, the coupling developed in Section 1.1.

Why embodied tasks are partially observed

That clean MDP contract rests on one assumption that no physical robot ever satisfies, and dropping it forces a richer tuple.

The MDP assumes the agent sees \(s_t\). A robot never does. It sees pixels, joint encoders, force readings, and a language instruction, none of which recover full object pose, mass, friction, or another agent's intent. The honest contract is therefore a partially observable MDP (POMDP), the tuple \((\mathcal{S},\mathcal{A},\mathcal{O},P,Z,r,\gamma)\), which adds an observation space \(\mathcal{O}\) and an observation function

$$Z(o \mid s',a) = \Pr(o_{t+1}=o \mid s_{t+1}=s',\, a_t=a),$$

the probability of receiving observation \(o\) after action \(a\) lands the world in state \(s'\). Because \(Z\) is generally many-to-one and noisy, the observation \(o_t\) is a lossy projection of the state, not the state itself.

Why this matters on physical hardware: every real sensor discards information. A depth camera sees reflected photons, not object mass or friction coefficient. A joint encoder measures shaft angle, not contact force or link flex. Network latency and analog-to-digital converter (ADC) quantization add noise after the sensor captures the physical signal. Even a perfect sensor would not help, because a robot's compute budget limits how many signals it can read and process within one control cycle. So \(Z\) is unavoidably many-to-one: distinct states (object poses that differ by a millimeter, surfaces with different friction) emit observations that are statistically indistinguishable. A policy that ignores this conflates states it cannot separate, and it behaves suboptimally in exactly the situations where precision matters most. A robot that assumes its sensor reading is the world is not perceiving reality; it is perceiving a rumor about reality.

How \(Z\) works mechanically: think of it as a generative sensor model. Given the true next state \(s'\) and the last action \(a\), \(Z\) first projects \(s'\) through a forward sensor model (ray-casting for lidar, a projection matrix for a camera). It then adds noise whose parameters match the physical sensor: Gaussian additive noise for encoders, structured dropout for depth cameras at reflective surfaces. The action \(a\) enters because actuator motion during the sensing window can blur or shift the reading. Probabilistic filters (Kalman, particle) implement exactly this generative view. They run \(Z\) in reverse, weighting hypothesized states by how well each one explains the observation the agent actually received.

A POMDP policy cannot be Markov in \(o_t\); it must act on a sufficient statistic of history, the belief state \(b_t(s) = \Pr(s_t = s \mid o_{0:t}, a_{0:t-1})\), which is updated by Bayes' rule through \(Z\) and \(P\). The belief MDP over \(b_t\) is the formally correct object, and it is why memory, filtering, and recurrent or transformer policies appear the moment a task is genuinely partially observed. In practice, partial observation multiplies the information a policy must carry: a fully observed MDP policy needs to encode 1 timestep of input to act optimally, while the same task as a POMDP may require 50 or more past observations before the belief is sharp enough to distinguish two confusable states.

A note on what follows: the algorithm below already tracks cost budgets and a Lagrangian multiplier update, machinery that belongs to the constrained MDP (CMDP) formalized later in this section under "Embodied tasks are almost always constrained." It is included here so the full interaction loop, observation, belief update, reward, and cost, can be read as one piece; the cost terms \(c_i\), \(d_i\), and \(\lambda_i\) will make more sense once that section defines them formally.

Action spaces are a design decision, not a given

The same physical robot arm, same task, same hardware, but with the action space changed from raw joint torques to discrete skill primitives, can require an order of magnitude fewer training steps to reach the same success rate. Why does a labeling choice in a configuration file change how fast a robot learns?

Three shapes for an action space

The set \(\mathcal{A}\) is chosen, and the choice determines what can be learned. Common forms: a discrete menu (\(\mathcal{A}=\{1,\dots,n\}\), e.g. a fixed set of grasp primitives); a continuous motor space (\(\mathcal{A}\subseteq\mathbb{R}^d\), e.g. joint torques or end-effector velocities, almost always box-bounded (clipped to a fixed minimum and maximum on each dimension) by actuator limits); and a hierarchical space, where a high-level policy emits a subgoal or skill index and a low-level policy emits the motor command that realizes it, factoring \(\pi = \pi_{\text{hi}} \circ \pi_{\text{lo}}\). The same physical robot can be posed as a discrete-action problem (pick which skill) or a continuous-control problem (emit the torque), and these are not the same learning problem: they differ in exploration cost, sample complexity, and which failures are even representable. In one reported manipulation benchmark (results of this kind vary considerably by task and simulator, and should be read as illustrative of the effect's direction and rough magnitude rather than a fixed constant), switching a pick-and-place task from raw joint torques to a library of six grasp primitives reduced the number of training episodes needed to reach 80% success from roughly 50,000 to around 300, because the policy no longer needed to rediscover grasping from scratch on every run.

The signal for choosing between forms is the ratio of task horizon to skill duration. When a single high-level intention takes thousands of low-level timesteps (navigate a building floor, assemble a multi-part object), motor-level continuous control must credit reward across a horizon so long that gradient estimates go noisy and exploration becomes intractable (crediting reward across a horizon means tracing a success or failure back to which of the many earlier actions actually caused it; the more steps between action and outcome, the noisier that trace becomes). Discrete and hierarchical spaces compress that horizon: the high-level policy reasons over tens of skill invocations, not thousands of motor commands. But when sub-millimeter force or velocity control decides success (contact-rich manipulation, balancing), a menu of primitives discards exactly the resolution the task needs.

A concrete rule of thumb: start with continuous control when the task's success is sensitive to sub-centimeter or sub-Newton variation; add a discrete or hierarchical layer when the planning horizon exceeds roughly 200 low-level steps without a natural subgoal structure. Boston Dynamics' Atlas whole-body controller, for instance, operates at 1 kHz torque commands but is commanded by a higher-level footstep planner that emits targets at roughly 10 Hz, a two-level hierarchy that keeps each layer's horizon tractable.

Embodied tasks are almost always constrained

Choosing the action space decides what the policy can do; the next ingredient decides what it must never do, no matter how much reward beckons.

An unconstrained objective lets the optimizer spend anything to gain reward, including force, energy, and collision risk a real platform cannot afford. The contract that captures hard limits is the constrained MDP (CMDP) of Altman: an MDP augmented with one or more cost functions \(c_i(s,a)\ge 0\) and budgets \(d_i\). The agent maximizes return subject to a bound on expected discounted cost,

$$\max_{\pi}\; J(\pi) \quad \text{subject to} \quad J_{c_i}(\pi) = \mathbb{E}_{\tau\sim\pi}\!\left[\sum_{t=0}^{\infty}\gamma^{t} c_i(s_t,a_t)\right] \le d_i \quad \text{for each } i.$$

The distinction from a reward penalty is structural, not cosmetic. Folding cost into reward as \(r - \lambda c\) commits to a single exchange rate \(\lambda\) chosen before training; the optimizer is then free to buy back any amount of safety violation whenever \(\lambda c\) is cheaper than the reward gained. The CMDP keeps \(c\) as a separate accounting line with its own budget \(d\), so "collide less than \(d\) times in expectation" is a commitment the optimizer must respect rather than a price it may pay.

This is why later chapters reach for Lagrangian methods, which adapt \(\lambda\) to enforce the budget, and constrained policy optimization rather than a hand-tuned penalty.

The contract as runnable code

The environment below instantiates the full contract for a deliberately small embodied task: a point effector on a line must reach a goal while a wall it cannot pass through sits between some states and that goal. State is the true effector position; the agent never sees it directly and receives only a noisy position reading, making the task a POMDP. The action is a bounded continuous velocity command.

Reward is shaped progress toward the goal; collision with the wall is logged as a separate cost, not subtracted from reward, making the task a CMDP. The episode terminates on reaching the goal and truncates at a step limit.

# A minimal embodied task contract as a Gymnasium-style environment:
# 1D reach-to-goal under partial observation (noisy position) with a separate collision cost.
# obs != state (POMDP); reward and cost are distinct fields (CMDP).
import numpy as np
import gymnasium as gym
from gymnasium import spaces


class Reach1D(gym.Env):
    metadata = {"render_modes": []}

    def __init__(self, obs_noise=0.02, wall_x=0.5, max_steps=100):
        super().__init__()
        self.obs_noise = obs_noise      # std of the position sensor; obs is a lossy view of state
        self.wall_x = wall_x            # impassable wall the effector must not push through
        self.max_steps = max_steps
        self.goal = 0.9
        # Continuous, box-bounded action: a velocity command clipped to actuator limits.
        self.action_space = spaces.Box(low=-0.1, high=0.1, shape=(1,), dtype=np.float32)
        # Observation is the noisy reading plus the goal; it is NOT the true state x.
        self.observation_space = spaces.Box(
            low=np.array([0.0, 0.0], dtype=np.float32),
            high=np.array([1.0, 1.0], dtype=np.float32),
        )

    def _obs(self):
        noisy = self.x + self.np_random.normal(0.0, self.obs_noise)
        return np.array([np.clip(noisy, 0.0, 1.0), self.goal], dtype=np.float32)

    def reset(self, *, seed=None, options=None):
        super().reset(seed=seed)
        self.x = 0.1                    # true state, hidden from the agent
        self.steps = 0
        return self._obs(), {"true_x": self.x}

    def step(self, action):
        a = float(np.clip(action, self.action_space.low, self.action_space.high)[0])
        prev_x = self.x
        proposed = np.clip(self.x + a, 0.0, 1.0)
        # The wall is impassable: a move that would cross it is blocked and incurs a cost.
        crossed = (prev_x - self.wall_x) * (proposed - self.wall_x) < 0
        cost = 1.0 if crossed else 0.0
        self.x = prev_x if crossed else proposed

        dist = abs(self.goal - self.x)
        reward = (abs(self.goal - prev_x) - dist)   # shaped progress; cost is NOT folded in
        terminated = dist < 0.02
        if terminated:
            reward += 1.0
        self.steps += 1
        truncated = self.steps >= self.max_steps
        info = {"cost": cost, "true_x": self.x}      # cost kept on its own accounting line
        return self._obs(), float(reward), terminated, truncated, info


if __name__ == "__main__":
    env = Reach1D()
    obs, info = env.reset(seed=0)
    total_reward, total_cost = 0.0, 0.0
    for _ in range(env.max_steps):
        action = env.action_space.high           # naive: always push toward the goal
        obs, reward, terminated, truncated, info = env.step(action)
        total_reward += reward
        total_cost += info["cost"]
        if terminated or truncated:
            break
    print(f"return={total_reward:.3f}  collision_cost={total_cost:.0f}  reached={terminated}")