Section 9.5: Benchmark environment map

"A benchmark is a contract. Sign it the same way every time, or the results are not comparable."

A Referee Environment, Strictly

Benchmarks Are Constructs, Not Leaderboards

Ask yourself: when a leaderboard says a grasping policy scores 91%, ninety-one percent of what, measured by whose contact solver, on which object meshes, under what success threshold? A benchmark environment never measures "grasping" in the abstract; it measures one frozen construct: locomotion stability, contact-rich manipulation, multi-agent coordination, navigation under partial observability, household task completion, visual grounding, or sim-to-real robustness. The benchmark's application programming interface (API), assets, physics, sensor model, and metric define that construct, and the score is meaningless until you can name it.

The practical question is not which environment is most popular. The question is which environment makes the right failure possible. Consider one documented case. A team scored 91% on a manipulation benchmark, then saw only 40% success on the real robot. The benchmark's contact model did not reproduce the target gripper surface. The same policy and the same hardware produced a 51-percentage-point drop, which makes the reality gap measurable and traces it entirely to a missing construct. This is called the construct-validity gap, and, in practice, a benchmark comparison that ignores it is typically comparing noise rather than a genuine method difference. A policy that scores well in a simulator whose construct does not match the real task is not a validated policy: it is a well-rehearsed assumption. The team lacked a construct-matched friction probe, so it ran 50,000 rollouts confirming a score the real robot could never reproduce. A single 300-episode probe on the calibrated construct would have surfaced the failure before any policy tuning began. A grasping benchmark that never models contact slip cannot validate contact robustness. A household benchmark without persistent object state cannot validate long-horizon interaction. Figure 9.5B traces this reasoning as a flow: a claim's required features are matched against each environment's coverage, producing either a candidate environment or a logged claim limit.

Environment Families And Valid Claims

Gymnasium-style APIs are useful for clean reinforcement-learning contracts. PettingZoo extends that discipline to multi-agent settings. MuJoCo, robosuite, ManiSkill, and Isaac Lab are often chosen for contact, manipulation, locomotion, vectorized rollouts, and controller integration. Habitat, AI2-THOR, ProcTHOR, BEHAVIOR, and OmniGibson emphasize embodied navigation, household semantics, object state, and visual interaction.

Each family has a validity envelope. If the claim is about torque-level transfer, check dynamics, contacts, actuation, and timing. If the claim is about embodied household reasoning, check scene diversity, affordances, persistent state, and task definitions. If the claim is about policy comparison, check whether all methods run through the same wrapper and metric script.

Worked Example

Code Fragment 9.5.1 scores candidate benchmark families against a task claim. The point is not to rank tools globally. The point is to make the construct match explicit before the experiment starts.

# Match benchmark families to the construct a claim needs.
# Weak matches become limits on what the result can support.
claim_needs = {"contact", "object_state", "vision", "failure_labels"}
benchmarks = {
    "MuJoCo manipulation": {"contact", "controller_timing", "failure_labels"},
    "ManiSkill": {"contact", "vision", "object_state", "failure_labels"},
    "Habitat navigation": {"vision", "layout_diversity", "navigation_metrics"},
    "ProcTHOR household": {"vision", "object_state", "layout_diversity"},
}

for name, supports in benchmarks.items():
    missing = sorted(claim_needs - supports)
    verdict = "candidate" if not missing else f"claim limit: missing {missing}"
    print(f"{name}: {verdict}")

MuJoCo manipulation: claim limit: missing ['object_state', 'vision']
ManiSkill: candidate
Habitat navigation: claim limit: missing ['contact', 'failure_labels', 'object_state']
ProcTHOR household: claim limit: missing ['contact', 'failure_labels']

Practical Recipe

1. Write the claim as a construct: contact robustness, visual navigation, household state tracking, coordination, or transfer.
2. Choose an environment family whose observation, action, asset, reset, and metric contract can measure that construct.
3. Run a random policy, a scripted baseline, and the intended policy through the same wrapper to test logging and metric sanity.
4. Record failures as structured cases: perception, state, planning, control, task semantics, timing, or metric mismatch.
5. Report the benchmark's unsupported assumptions beside the positive result.

Naming those seven fields in a self-check is only the first half of the discipline; the second half is binding them into a contract the whole evaluation loop must honor. Benchmark environment map becomes useful when it is tied to a closed-loop contract. The contract names the observation stream, action representation, reset distribution, timing budget, metric, assets, version, and evaluation artifact. Without that contract, a model can look capable in a benchmark table while failing the first time the real task changes an object state the environment never modeled.

The graduate-level habit is to separate three claims. The benchmark claim says what construct the environment measures. The systems claim says what wrapper and artifact make the result reproducible. The transfer claim says which real-world assumptions remain untested. Keeping those claims separate prevents benchmark convenience from becoming benchmark overreach.

Before reading on, guess: if two researchers both claim to use "ManiSkill" but one runs version 2.0 and the other runs version 3.0 with different object meshes and friction defaults, can their benchmark scores be directly compared? The answer determines whether the numbers in a published table are evidence or noise.

Write The Card Before The Policy

A robust implementation starts with a benchmark card before it starts with a policy. The card should log the environment version, wrapper stack, task panel, observation and action spaces, seeds, metric script, asset set, and failure taxonomy. The policy result is interpretable only after that card exists.

Why this matters for embodied AI: a real robot carries physical tolerances, and a simulator approximates those tolerances differently depending on its version, asset set, and reset distribution. Without a written card, two researchers may both call their setups "ManiSkill" yet run different object meshes, friction coefficients, and success thresholds. The gap between their scores then looks identical to a genuine policy difference, so every hardware deployment decision rests on unreliable evidence.

How it works: the card acts as a hash over the environment contract. Each field pins one degree of freedom, so any change yields a different card. Two policies sharing an identical card come from the same frozen world and compare cleanly; any unspecified field is an uncontrolled variable that can mask a real performance difference.

1. Write a benchmark card with construct, environment version, wrapper stack, seeds, metric, and failure labels.
2. Run random and scripted baselines to catch broken resets, hidden privileged state, and metric leakage.
3. Freeze the held-out task panel before tuning policies or curricula.
4. Save videos, traces, configs, metric outputs, and failure labels in one artifact bundle.
5. Compare methods only when one script evaluates them on the same benchmark card.

# Build a benchmark card before reporting a policy score.
# The card states what construct the environment can validly measure.
from dataclasses import dataclass, asdict

@dataclass
class BenchmarkCard:
    environment: str
    construct: str
    observation_space: str
    action_space: str
    metric: str
    known_limit: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

card = BenchmarkCard(
    environment="ManiSkill pick-and-place panel",
    construct="vision-conditioned contact manipulation",
    observation_space="RGB-D camera plus proprioception",
    action_space="end-effector delta pose",
    metric="held-out object success rate with failure labels",
    known_limit="requires real friction calibration before deployment claims",
)
print(card.as_row())

{'environment': 'ManiSkill pick-and-place panel', 'construct': 'vision-conditioned contact manipulation', 'observation_space': 'RGB-D camera plus proprioception', 'action_space': 'end-effector delta pose', 'metric': 'held-out object success rate with failure labels', 'known_limit': 'requires real friction calibration before deployment claims'}

Expected output: the record exposes the benchmark's construct and its known limit before any model result appears. That ordering keeps the environment from being used to support claims it cannot measure.

Even with the card written first, a policy will sometimes score poorly, and the same construct discipline that selected the environment now tells you how to read that failure. When a benchmark experiment fails, avoid labeling the method as weak before checking the wrapper and construct. First assign the failure to perception, state estimation, planning, control, task semantics, timing, data coverage, or metric mismatch. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing benchmark score into a reusable diagnostic asset.