Section 2.1: Agents and environments formally

"The environment is the part of the experiment that gets a vote after every action."

A Boundary-Conscious Embodied AI Agent

Two teams run the identical policy on the identical robot and report success rates 30 points apart; neither team is lying, and the only thing that differs is how their environment wrapper labeled the moment an episode ended. That is what happens when the agent-environment loop stays an informal sketch instead of a precise object you can specify, implement, test, and log. The point is not to admire a loop diagram. The point is to know exactly what happens when a robot receives a sensor packet, chooses an action, waits for the world to respond, and records evidence about the result. Figure 2.1A labels the four typed signals (observation \(o_t\), action \(a_t\), next state \(s_{t+1}\), reward \(r_t\)) that cross the agent-environment boundary on every timestep, and Figure 2.1 recasts that same loop as the evidence, decision, consequence pattern used throughout this section.

The distinction matters because many embodied failures are interface failures, not policy failures (the Theory section below defines "terminated" and "truncated" precisely; for now, read truncation as an external cutoff, such as a time limit, that ends an episode before the task itself finished). A policy may be competent, but the environment wrapper may hide time-limit truncation. A simulator may expose privileged state that the real robot never observes. A logger may record reward yet omit the action clipping that changed the actual command. Consider illustrative figures consistent with documented evaluation artifacts in legged locomotion benchmarks (circa 2023-2024). The same policy scores near 90% success when a wrapper logs truncated episodes separately, and near 54% when the wrapper silently counts them as failures. Interface ambiguity alone opens that gap of more than 30 percentage points; nothing about the agent changed.

Theory

By the end of this Theory section you will be able to write down, for any embodied system, the exact fields that cross the agent-environment boundary at each timestep (observation, action, reward, terminated, truncated, info) and explain why the reset/step contract, not a fixed input stream, defines that boundary. That specification skill, not the vocabulary alone, is what the Practical Recipe and audit code below put to use.

At time \(t\), an agent receives an observation \(o_t\), chooses an action \(a_t\), and receives a consequence that usually includes a new observation \(o_{t+1}\), a reward or score \(r_{t+1}\), and episode status. The environment owns the transition dynamics. The agent owns the decision rule. The evaluator owns the claim about whether behavior was good.

For a fully specified single-agent environment, the minimal contract is close to the Gymnasium pattern: reset starts an episode and returns an initial observation plus metadata; step(action) advances the world and returns observation, reward, terminated, truncated, and info. Terminated means the task reached a natural end. Truncated means an external limit, such as a time limit, stopped the episode.

Why transition dynamics matter in embodied AI. A physical robot cannot pause the world while computing its next action. Motors have inertia, joints have compliance, and surfaces deform under load. If the transition function is wrong or hidden, a policy trained in simulation may issue commands that the real actuator cannot execute within one control cycle. That mismatch causes drift, and the drift compounds across every subsequent step. A 2% per-step position error sounds trivial. Over a 200-step episode it accumulates into a robot nearly 50 cm off course by the final action. This is why the same policy can look flawless in a simulator that returns perfect state and catastrophic on the physical platform that returns noisy, delayed sensor packets. Getting the dynamics right is not a formalism exercise; it is what separates a policy that transfers to hardware from one that works only in the log file.

How the transition function works. At each timestep, the environment applies the agent's action to its internal state using the rule \(s_{t+1} \sim T(\cdot \mid s_t, a_t)\), then filters that state through an observation function to produce \(o_{t+1}\). In code, step(action) performs both operations: it updates internal physics or state variables, then packages the result into the typed fields the agent receives. Logging the raw state alongside the observation lets engineers audit where the gap between what the robot sees and what actually happened first appeared.

Everything so far assumed a single decision process owns every action, but the moment a second agent enters the scene that assumption needs explicit machinery. For multi-agent settings, the contract also needs turn order or simultaneous actions. PettingZoo makes that distinction explicit through sequential and parallel APIs. This matters for embodied systems with people, other robots, traffic participants, or adversarial agents.

The single-agent contract holds when one decision process controls all actions and no other agent's behavior alters the transition function. It breaks the moment another agent's policy becomes part of the dynamics: a delivery robot sharing a corridor cannot treat human motion as fixed noise, because human paths respond to its trajectory. The interface must then expose whose turn it is, which agents are active, and whose observation carries evidence of other agents' recent actions. Heuristic: if removing one agent would change another's optimal policy, the single-agent contract is wrong.

Worked Example

The fastest way to see that typed transition boundary become concrete is to build the smallest possible version of it by hand. Code Fragment 2.1.1 builds a tiny environment contract without any reinforcement learning library. The example is deliberately small so that every returned field is visible.

# Section 2.1: runnable checkpoint for Environment dynamics and transition functions.
# Keep the output small so the evidence record can be inspected directly.
from dataclasses import dataclass

@dataclass
class Transition:
    observation: dict
    action: str
    reward: float
    terminated: bool
    truncated: bool
    info: dict

def step(position, action, time_step):
    delta = 1 if action == "right" else -1
    next_position = position + delta
    reached_goal = next_position >= 3
    timed_out = time_step >= 4
    reward = 1.0 if reached_goal else -0.01
    return Transition(
        observation={"position": next_position},
        action=action,
        reward=reward,
        terminated=reached_goal,
        truncated=timed_out and not reached_goal,
        info={"latency_ms": 12, "action_clipped": False},
    )

print(step(position=2, action="right", time_step=3))

Practical Recipe

1. Name the agent process and the environment process separately.
2. Specify reset, observation, action, reward, termination, truncation, and info fields before training.
3. Track latency for observation capture, policy inference, transport, and action execution.
4. Log the full transition tuple before aggregating success rate or return.
5. Run a no-op or safe-action baseline to verify that the environment boundary behaves as expected.

The formal interface is useful only if it prevents silent changes in the experiment. A policy result should say which environment version produced it, how reset sampled initial state, which action space was accepted, whether the command was clipped, why the episode ended, and whether the ending was a task termination or an external truncation.

The most common mistake is to treat the environment as background code. In embodied AI the environment is part of the scientific claim. If a wrapper changes observations, action limits, time limits, rewards, or diagnostic info, it changes the meaning of the result.

Before training, run an interface audit on one transition. The audit should fail if a required field is missing, if termination and truncation are confused, or if action clipping is hidden.

1. Declare the reset output and step output as named fields.
2. Check that observation and action types match the declared spaces.
3. Log termination and truncation as different fields.
4. Put latency, clipping, safety gate status, and wrapper version in info.
5. Save the first transition from every experiment run as a smoke-test artifact.

# Audit one transition record before trusting an environment result.
transition = {
    "observation": {"position": 3},
    "action": "right",
    "reward": 1.0,
    "terminated": True,
    "truncated": False,
    "info": {"latency_ms": 12, "action_clipped": False, "wrapper_version": "v2"},
}

def audit_transition(row: dict[str, object]) -> list[str]:
    required = {"observation", "action", "reward", "terminated", "truncated", "info"}
    problems = [f"missing {key}" for key in sorted(required - row.keys())]
    if row.get("terminated") and row.get("truncated"):
        problems.append("terminated and truncated cannot both explain the same ending")
    info = row.get("info", {})
    for key in ["latency_ms", "action_clipped", "wrapper_version"]:
        if key not in info:
            problems.append(f"info missing {key}")
    return problems

print(audit_transition(transition))

When an agent-environment experiment fails, first inspect the transition boundary. Check reset distribution, action clipping, wrapper order, time-limit handling, reward emission, and diagnostic info before changing the policy.