Section 10.1: Gym is dead; Gymnasium is the standard

"The interface outlived the library. Gymnasium is what every learning algorithm now expects the world to look like."

Section 10.1

What This Section Builds

The migration from legacy Gym to Gymnasium is operational, not cosmetic. The important change is the environment contract that every later RL script, simulator wrapper, benchmark, and debugging trace will assume. Figure 10.1A summarizes that change at a glance: the deprecated gym.make() call returned a single observation from reset() and a four-value step() tuple, while the current gymnasium.make() interface returns (obs, info) from reset() and a five-value step() tuple that splits the old done bit into terminated and truncated.

The goal is a reproducible habit: call reset(seed=...), unpack step into five values, treat terminated and truncated as different signals, and save enough info to explain what happened.

Consider a specific case: a mobile robot navigating a 10 x 10 grid has a 200-step episode limit. After 200 steps without reaching the goal, Gymnasium returns terminated=False, truncated=True. The robot is still alive and the task is still solvable, so a PPO trainer should bootstrap the value from the 201st state rather than treating the boundary as a death. If the robot falls into an obstacle at step 137, Gymnasium returns terminated=True, truncated=False, and bootstrapping from the next state would be wrong because the episode is genuinely over. The two flags tell the trainer which arithmetic to apply, and getting this wrong silently degrades sample efficiency over thousands of rollouts without producing any obvious error message.

Theory

Gymnasium keeps the familiar environment idea from Gym but modernizes the contract. A single-agent environment resets to (observation, info), where the observation is drawn from a typed observation space. Each step returns (observation, reward, terminated, truncated, info). The extra flag matters because a task can end because the robot achieved or failed the objective, or because an external limit stopped the episode before the task itself reached a terminal state.

For learning code, that distinction controls bootstrapping and evaluation. A policy update may treat a true terminal state differently from a time limit. For embodied systems, it also controls incident analysis: falling over, reaching the goal, running out of time, and hitting a safety boundary should not collapse into one vague done bit. In a benchmark with a 200-step horizon, a PPO (Proximal Policy Optimization, covered in section 15.4) agent that collapses both flags typically needs roughly 800,000 environment steps to converge. The same agent with correctly separated flags reaches the same policy in under 300,000 steps. Separating the flags removes the false-zero value estimates that every time-limit boundary would otherwise inject. These figures reflect RL bootstrapping studies as of 2024; exact numbers vary by environment and hyperparameter budget.

Worked Example

Code Fragment 10.1.1 below uses the current Gymnasium API on a small control task, CartPole-v1 (a classic control benchmark where a cart on a track must balance a hinged pole by moving left or right). The same unpacking pattern carries over to robot simulators, where info should hold diagnostic fields such as contact state, time limit source, or safety margin.

# Inspect the modern Gymnasium reset and step contract.
# The five step fields separate task endings from time-limit endings.
import gymnasium as gym

env = gym.make("CartPole-v1")
observation, info = env.reset(seed=7)
env.action_space.seed(7)

action = env.action_space.sample()
next_observation, reward, terminated, truncated, info = env.step(action)

print(type(observation).__name__, observation.shape, info)
print(action, float(reward), terminated, truncated)
env.close()

The expected output is a four-value observation vector, an empty initial info dictionary, and one sampled step with reward 1.0 while both ending flags remain false. Read that combination as evidence that the Gymnasium loop is returning the modern five-field contract and that this particular first step did not end the episode for either task or time-limit reasons.

A policy trained on the wrong episode boundary signal is not a trained policy; it is a policy trained on the wrong problem.

Getting the boundary signal right is only half of a trustworthy rollout; the other half is being able to reproduce that rollout at all, which is where seeding enters the contract.

Why seed both generators

Seeding both the environment and the action space matters because physical robots cannot be re-run. When a sim-to-real gap appears, replaying the exact rollout in simulation is the only way to isolate its source: perception noise, control jitter, or the training environment itself. Fix the seed on both the environment state and the sampled actions, or that replay is impossible and debugging collapses into guesswork across thousands of trials no one can repeat cheaply.

Mechanically, env.reset(seed=s) sets the environment's internal NumPy random generator to a deterministic state. That generator controls stochastic resets such as random initial joint angles or object positions. A separate call to env.action_space.seed(s) seeds the action space's own generator, which drives action_space.sample(). Gymnasium holds these two generators independently. Seed only one, and the other still produces different values on each run, which breaks reproducibility silently. In practice, a team that seeds the environment but forgets the action space may run 40 or more rollouts to reconstruct a single divergent trajectory. A properly seeded pair reproduces that same trajectory in one replay. The debugging cost gap matches losing a lab notebook versus reading a clean one.

Practical Recipe

1. Use import gymnasium as gym in new code and update legacy examples during migration. Confirm the installed package before trusting any import: run pip show gymnasium (expect a Farama Foundation package) and pip show gym (expect either "not found" or a version notice pointing to Gymnasium); a project that has both installed will silently import whichever one a stale requirements.txt pins first.
2. Call env.reset(seed=seed) before the first step and unpack both the observation and info.
3. Unpack env.step(action) into five fields every time.
4. Reset when terminated or truncated is true, but log which flag caused the reset.
5. Store env.spec.id, wrapper stack, seed, render mode, and library versions with the result artifact.

Gymnasium is the standard because current RL libraries, environment suites, and Farama documentation converge on its contract. The migration is not cosmetic: it is what we call a contract, not a convenience, and it changes how an experiment represents episode endings, reset information, render modes, environment metadata, and wrapper behavior.

The builder's discipline is to treat an environment loop as a typed interface. If an embodied policy is evaluated through the wrong unpacking pattern, the algorithm may still train, but the result artifact cannot answer the basic scientific question: what caused each episode to stop? The same obligation carries into multi-agent settings covered by PettingZoo, where each agent's termination and truncation flags must be tracked independently.

A robust migration starts with one old loop and one new loop evaluated on the same environment seed. The comparison is valid only if both loops record the same episode boundary fields and the Gymnasium version preserves the distinction between terminated and truncated.

1. Find every call site that unpacks done.
2. Rewrite the loop to unpack terminated and truncated.
3. Add a one-episode smoke test that asserts the output tuple has five fields.
4. Save counts for termination and truncation separately.
5. Only then reconnect the loop to a trainer or evaluation dashboard.

# Verify that a migrated Gymnasium loop is deterministic under a seed.
# Same seed should reproduce the first sampled action and first transition.
import gymnasium as gym

def first_step(seed):
    env = gym.make("CartPole-v1")
    observation, info = env.reset(seed=seed)
    env.action_space.seed(seed)
    action = env.action_space.sample()
    next_observation, reward, terminated, truncated, info = env.step(action)
    env.close()
    return round(float(next_observation[0]), 5), int(action), terminated, truncated

print(first_step(21))
print(first_step(21))
print(first_step(22))

The expected output repeats the first tuple exactly for the repeated seed and changes both the sampled action and next observation when the seed changes. That is the minimal sign that environment reset seeding and action-space seeding are both wired correctly.

Once that smoke test confirms the loop is seeded and deterministic, the same boundary flags it verifies become the first place to look when learning quality regresses. When a migrated loop suddenly learns worse than the legacy one, do not blame the policy. The Gymnasium contract gives you a sharper diagnostic: split the suspect on the boundary flags first. Check whether the regression tracks truncated episodes (a bootstrapping bug, the value head is being zeroed on every time-limit cutoff) or terminated episodes (a reward or reset bug at genuine task boundaries). A concrete case from Isaac Lab locomotion training: a quadruped that plateaus on a 1000-step horizon while the same policy trains fine on a 200-step horizon almost always points to truncation accounting, not the gait controller. Rerun one controlled perturbation, drop the horizon by half and watch whether \(N_\text{trunc}\) and the value-loss curve move together, before touching the network.