Section 10.6: Evaluation protocol and seeding

"A number without a seed policy and an episode budget is a story, not a measurement."

Section 10.6

What This Section Builds

Swap one integer in your evaluation script and a "broken" locomotion policy can suddenly post a state-of-the-art success rate, with no retraining and no code change: that integer is the seed, and an evaluation protocol is the set of guardrails that stops it from silently rewriting your conclusions. A seed policy controls the starting randomness; an evaluation protocol controls which tasks, wrappers, metrics, and failure labels are compared.

The goal is one auditable comparison artifact. If two methods are compared, they should be evaluated in one pass on the same environment panel, same wrapper stack, same seed list, and same metric definitions.

Theory

Seeding is not a guarantee that every library, process, and physics engine will behave identically across machines. It is a contract for controlled comparison: the experiment declares how initial conditions, action sampling, environment randomness, and evaluation panels are generated.

Gymnasium supports this discipline by passing a seed into reset and allowing the action space to be seeded for reproducible sampling. A strong protocol records both, then evaluates methods under the same seed list instead of reporting numbers from separate runs. Figure 10.6B shows how this protocol flows from a frozen seed panel to a single comparison artifact.

A panel matters for embodied AI because physical contact, gravity, and joint-angle initialization interact nonlinearly. One lucky starting posture can make a brittle locomotion policy look robust, while a single unlucky terrain seed can hide a strong one. Policies that will run on hardware must prove reliability across a distribution of starts, not just a favorable singleton. A panel forces that proof into the evaluation instead of leaving it to the reader's interpretation. A policy that looks reliable on one seed and crumbles on thirty is not a robust policy; it is a well-placed bet.

In practice, a seed panel is a fixed list of integers declared before any rollout begins. Each integer seeds both env.reset() and env.action_space.seed(), so the starting observation and any random baseline actions are jointly controlled. Panel size is chosen to keep the standard error of the mean (SEM, the estimated standard deviation of the sample mean, which shrinks as \(1/\sqrt{k}\) with panel size \(k\); see the Lab below for a hands-on derivation of this shrinkage) metric below a target threshold; 30 to 50 seeds is typical for continuous-control benchmarks. To see why size matters, consider that a single-seed evaluation has an illustrative, order-of-magnitude chance, roughly 20% for a typical return distribution, of landing in the top or bottom quartile of the start distribution by pure luck, while a 30-seed panel typically reduces that misclassification probability well below 1%, though the exact figures depend on the shape of the underlying return distribution rather than being a universal guarantee. The list is frozen, version-controlled alongside the environment config, and never extended post-hoc to favor a particular method.

Worked Example

Code Fragment 10.6.1 demonstrates a minimal seed smoke test. The same seed reproduces the first sampled action and transition; a different seed changes the trace.

# Check whether the environment and action sampling are seeded together.
# The same seed should reproduce the first action and first transition.
import gymnasium as gym

def first_step(seed):
    env = gym.make("CartPole-v1")
    observation, info = env.reset(seed=seed)
    env.action_space.seed(seed)
    action = env.action_space.sample()
    next_observation, reward, terminated, truncated, info = env.step(action)
    env.close()
    return round(float(next_observation[0]), 5), int(action), terminated, truncated

print(first_step(21))
print(first_step(21))
print(first_step(22))

The expected output repeats the first trace exactly under the repeated seed, then changes when the seed changes. That is the correct interpretation for a seed smoke test: determinism within a seed, variation across seeds, and no hidden episode ending in any of the one-step traces.

Practical Recipe

Because Gymnasium hands you the seed hooks but leaves the protocol to you, the following recipe turns that responsibility into a fixed sequence of steps you can apply to any comparison.

1. Declare the seed list before running methods.
2. Evaluate every compared method on the same environment ids, wrappers, and seeds.
3. Save per-seed results before aggregating means or confidence intervals.
4. Report termination, truncation, and failure labels with the primary success metric.
5. Save one machine-readable artifact that contains all compared numbers.

Evaluation protocols prevent accidental storytelling. Without a shared protocol, one method can win on easier starts, looser time limits, missing wrappers, or a softer failure classifier, and the table still prints clean numbers that do not answer the claimed question. Call this the lucky-seed illusion, and the cost is concrete: two locomotion policies on separate 10-seed panels reported an 18-point success-rate gap that shrank below 3 points on a shared 50-seed panel (a recurring finding in continuous-control reproducibility work, e.g., Agarwal et al., 2021).

A robust implementation produces a method-by-seed table first, then derives summaries from it. This keeps each claim traceable to a concrete run under the same protocol.

1. Write a protocol config with environment id, wrappers, seed list, max steps, and metrics.
2. Loop over methods inside the same evaluation script.
3. Loop over seeds inside each method and write one row per episode.
4. Aggregate only after all raw rows are saved.
5. Audit every table number by recomputing it from the saved artifact.

# Build a small co-computed evaluation panel.
# Each method is evaluated on the same seed list in one artifact.
import gymnasium as gym

def rollout(method_name, seed):
    env = gym.make("CartPole-v1", max_episode_steps=5)
    observation, info = env.reset(seed=seed)
    env.action_space.seed(seed)
    total_reward = 0.0
    terminated = truncated = False

    while not (terminated or truncated):
        action = 0 if method_name == "always_left" else env.action_space.sample()
        observation, reward, terminated, truncated, info = env.step(action)
        total_reward += float(reward)

    env.close()
    return {"method": method_name, "seed": seed, "reward": total_reward, "truncated": truncated}

panel = [rollout(method, seed) for method in ["always_left", "random"] for seed in [1, 2]]
print(panel)

The expected output is one co-computed row per method and seed, not one summary per method. Even though the toy panel gives the same reward in all four rows, the important interpretation is that the evidence format makes a valid comparison possible because every row was produced under the same protocol.

Once that row-level artifact exists, it becomes the starting point for diagnosis rather than just scoring: the per-seed rows tell you not only that a policy underperformed but on exactly which starts it failed.

When a locomotion or manipulation policy underperforms on the seed panel, do not label the whole method as weak. First, assign the failure to a specific physical subsystem. Candidates include foot-contact timing on uneven terrain, joint-velocity estimation lag from a 20ms IMU filter delay, gripper slip from uncalibrated friction coefficients in MuJoCo's contact solver, or truncation from a 10-second horizon that fires before a Franka Panda arm completes its grasp. Next, rerun one controlled perturbation that isolates the suspected cause. Hold terrain randomization fixed, for example, and vary only the initial joint configuration across seeds. This pattern turns a disappointing rollout into a reusable diagnostic asset. It also tells you whether the failure is a seeding artifact or a genuine policy gap that will surface on hardware.