Section 10.2: Observation and action spaces

"The wrong space does not crash. It trains quietly for days and hands you a policy that was never solving your problem."

A Silent Type Error

What This Section Builds

Hand a policy a single number more than the real robot can sense, and it will happily learn to win on that number, leaving you with a sim trophy and a robot that fails the moment that number disappears. That number lives or dies in the observation and action spaces: the typed boundary where the simulator tells the learning code what can be sensed, what can be commanded, and which samples are invalid before the policy ever trains. Figure 10.2A shows this boundary for a continuous-control robot: a structured Dict observation space pairing an image with a proprioception vector, and a bounded Box action space carrying joint torque limits.

The goal is to make every observation and action inspectable. A camera tensor, proprioceptive vector, gripper command, and discrete mode switch should each have shape, dtype, bounds, and units that a teammate can verify.

Theory

A Gymnasium space is a mathematical set with enough metadata for code to sample, validate, batch, flatten, or reject values. Box handles continuous tensors, Discrete handles one categorical choice, MultiDiscrete handles several categorical choices, and Dict keeps structured observations readable instead of forcing every signal into one anonymous vector.

The design question is not "what can my simulator emit?" The better question is "what should the policy be allowed to observe and command?" That distinction prevents hidden state leaks, mismatched units, and action dimensions that a robot controller cannot execute. Getting this wrong is expensive. A policy that leaks ground-truth object pose from the simulator into the observation typically solves the task in under 200 training episodes. Restrict the same policy architecture to what the real robot actually senses, and it can need 10,000 episodes or more to converge, because it must now learn perception alongside control instead of skipping it.

Worked Example

Code Fragment 10.2.1 builds a structured observation space for a tabletop robot. The camera, arm joints, and gripper state stay separate, so debugging can identify which signal violated the contract.

# Define a structured observation space for a tabletop robot.
# Dict spaces keep images, joints, and gripper state inspectable.
from gymnasium import spaces

observation_space = spaces.Dict({
    "image": spaces.Box(0, 255, shape=(64, 64, 3), dtype="uint8"),
    "joint": spaces.Box(-1.0, 1.0, shape=(7,), dtype="float32"),
    "gripper": spaces.Discrete(2),
})

sample = observation_space.sample()
print(observation_space.contains(sample))
print(sample["image"].shape, sample["joint"].shape, sample["gripper"])

The expected output starts with True, which confirms that the sampled observation satisfies the declared Dict space exactly. The listed shapes then make the contract visible: one RGB image tensor, one seven-joint vector, and one binary gripper state.

Practical Recipe

1. Choose Dict for heterogeneous sensor packets instead of flattening too early.
2. Choose Box for bounded continuous vectors such as joint positions, velocities, images, or normalized actions.
3. Choose Discrete or MultiDiscrete for mode choices, buttons, symbolic skills, or gripper states.
4. Check observation_space.contains(obs) after reset and step while developing a custom environment.
5. Document units and scaling near the space declaration, not in a separate notebook.

Once every key passes that contract test, the next risk is not whether the space is well-formed but whether it honestly describes the real robot. Observation and action spaces are where simulation shortcuts often leak into results. If the observation includes exact object pose from the simulator while the real robot would only have pixels, this is called the privileged-state illusion, and the task has changed. If the action space accepts arbitrary forces while the real controller accepts joint targets, the policy may solve a fantasy control problem. A space that misrepresents the real robot is not an abstraction: it is a different problem wearing the same name.

Audit every space as a claim about sensing and actuation: what information is available, what commands are legal, and what transformations reshape the data before the learning algorithm sees it.

A robust implementation writes the spaces before writing the dynamics. That order forces the author to decide what the agent may know and do, then makes the simulator produce values that match the declared contract.

1. Declare the observation and action spaces in the environment constructor.
2. Run contains checks after every custom reset and step during development.
3. Keep simulator-only diagnostic variables in info, not in the policy observation.
4. Record space definitions in the experiment artifact, including units and scaling.
5. Change the policy architecture only after the environment contract is stable.

# Define action choices separately from continuous observations.
# This makes controller legality visible before training starts.
from gymnasium import spaces

action_space = spaces.MultiDiscrete([3, 3, 2])
action_space.seed(4)

action = action_space.sample()
meaning = {
    "x_motion": ["left", "hold", "right"][int(action[0])],
    "y_motion": ["back", "hold", "forward"][int(action[1])],
    "gripper": ["open", "close"][int(action[2])],
}

print(action.tolist())
print(meaning)
print(action_space.contains(action))

The expected output shows the integer action tuple and its physical decoding side by side, then confirms with True that the action is legal under the declared space. This is the interpretation readers should want in a robot log: not only what integers were sampled, but what motion command they meant.

A MultiDiscrete space matters in embodied AI because real actuators often have independent discrete modes. A mobile base may choose from three heading increments, a gripper from two states, and a wrist from four orientations: these are separate physical constraints with separate failure modes. Collapsing them into a single Discrete(24) forces the policy to learn that most of the 24 indices are illegal combinations. That wastes sample budget and makes safety violations harder to audit per joint. Informal benchmarks on 6-DoF arms suggest the cost is significant in practice, though the exact ratio varies with task and reward shaping: switching from a flat Discrete(729) space to a MultiDiscrete([3,3,3,3,3,3]) space has been observed to cut the episodes needed to reach a comparable success rate roughly fivefold, typically from tens of thousands down to a few thousand, because the policy no longer had to discover by trial and error which of the 729 indices were physically coherent commands.

Mechanically, MultiDiscrete([n_1, n_2, ..., n_k]) stores a 1-D integer array of length \(k\) where each element \(i\) is drawn independently from \(\{0, \ldots, n_i - 1\}\). The contains check verifies each element against its own bound independently, so a violation in one axis is localized rather than masked by valid values elsewhere.

Keeping each axis independently checkable pays off most when a run goes wrong and you have to find out which part of the interface betrayed you. When an experiment about observation and action spaces fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.