Section 12.1: Why standardized benchmarks matter

"A comparison is only as valid as the common ground it stands on. The benchmark is that ground."

Section 12.1

What This Section Builds

Change one wrapper from RGB-D to state observations (the ground-truth object poses and joint values the simulator tracks internally, rather than rendered images) and the same policy can swing 20 success points on the identical task, which is exactly how two papers end up reporting different numbers for the same capability. To stop that confusion, this section separates the construct being measured, such as dexterous manipulation or social navigation, from the harness that measures it, such as ManiSkill, robosuite, Habitat, Brax, or Isaac Lab.

The goal is a reproducible habit: define the task contract, freeze the evaluation split, run all methods through the same harness, and save one artifact that contains the configuration and every compared metric. As Figure 12.1A captures, a benchmark score earns trust only once the task panel, split, seeds, wrappers, and metric all line up; mismatched seeds and splits get set aside, not folded into the comparison.

Theory

We can view a benchmark as a sampled set of closed-loop episodes. For episode \(i\) and seed \(s\), the harness samples an initial state, exposes observations \(o_t\), accepts actions \(a_t\), and computes a score \(m_i\). The published number is usually an aggregate such as \(\frac{1}{N}\sum_i m_i\), so the meaning of the aggregate depends on which episodes entered the sum. Figure 12.1B traces this flow end to end, from the shared task panel through the harness to the per-seed scores and the single saved artifact, and marks the validity gate that every compared method must clear.

The practical design rule is to make the sampling contract explicit. A claim about general manipulation needs held-out objects, poses, scenes, or task families. A claim about control robustness needs held-out disturbances or physics parameters. A claim about benchmark speed, common in Brax and Isaac Lab comparisons, needs the same number of environments, observation modalities, rollout horizon, accelerator, and logging overhead.

Worked Example

Code Fragment 1 turns the same-config rule into a small audit. The point is not the dataclass itself, it is the habit of refusing to compare two rows unless the panel, split, seeds, wrappers, and metric match exactly.

# Check whether two benchmark rows belong in one comparison table.
# Paper-facing comparisons require the same panel, split, seed set,
# wrapper stack, and metric for every method being compared.
from dataclasses import dataclass

@dataclass(frozen=True)
class EvaluationRun:
    method: str
    panel: str
    split: str
    seed_set: tuple[int, ...]
    wrappers: tuple[str, ...]
    metric: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

baseline = EvaluationRun("BC", "pick-place-v1", "heldout_objects", (0, 1, 2), ("rgbd",), "success_rate")
candidate = EvaluationRun("DiffusionPolicy", "pick-place-v1", "heldout_objects", (0, 1, 2), ("rgbd",), "success_rate")

comparable = baseline.__dict__ | {"method": candidate.method}
same_config = comparable == candidate.__dict__
print(f"paper_table_ready={same_config}")

Practical Recipe

1. Name the construct: manipulation success, lifelong transfer, navigation efficiency, social compliance, or physics throughput.
2. Freeze the task panel, train/validation/test split, seed list, wrapper stack, simulator version, and metric before model selection.
3. Run a transparent baseline through the exact same evaluation script as the proposed method.
4. Report aggregate metrics with confidence intervals or seed-level values, not only a single best run.
5. Record failures as structured cases: perception, planning, contact dynamics, timing, language grounding, human interaction, or evaluation leakage.

Standardized benchmarks matter because embodied AI claims inflate accidentally: a manipulation policy memorizes training objects, a navigation agent tunes to public validation houses, a simulator-speed result shifts once rendering or logging enters the loop. The benchmark contract blocks these mistakes by fixing which variation training may see and which variation evaluation reserves. The stakes are concrete. Two teams evaluate on ManiSkill pick-place and report a 17-point gap, yet rerunning both under the same RGBD wrapper and identical seeds collapses it to 4 points, within seed noise. The benchmark did not change; the measurement contract did.

The graduate-level habit is to separate three claims, building on the construct/harness split introduced earlier in this section. The construct claim says what capability is measured. The harness claim says exactly how episodes are sampled and scored. The evidence claim says which same-config artifact supports the comparison. If those claims are mixed, a paper can appear to compare methods while actually comparing datasets, wrappers, or simulator settings. This confusion is so common it has a name: the protocol-mismatch problem, where numbers differ not because policies differ but because measurement contracts do, exactly the pattern already seen in the ManiSkill 17-point gap.

Every family in that table audits a split because the split is what the task panel concretely encodes, so it is worth examining that panel up close. What happens when a policy that scores 82% on familiar training objects meets a novel shape it has never seen? That question is the entire point of the task panel. The task panel is the explicit list of episodes, objects, scenes, or task variants that constitute one evaluation. Panel choice carries direct consequences on physical robots. A panel drawn from training objects lets a policy succeed by recognition rather than generalization. The number hides that distinction, but a novel geometry on real hardware exposes it at once. A policy that scores 82% on a familiar-object panel may score below 40% on a held-out panel. Hyperparameter tuning against the familiar panel typically cannot recover that gap, because the benchmark was measuring recognition, not the generalization capability the paper claims. Without the held-out split, 50 evaluation episodes on training objects produce zero failures and a confident-sounding number. Those same 50 episodes drawn from novel shapes expose the 42-point collapse instantly, revealing that the policy was recognizing, not generalizing.

How a Panel Is Built

Mechanically, you construct a task panel by partitioning variation along the axes that define the construct claim. A manipulation generalization claim splits variation across object geometry, texture, and pose. Training episodes draw from one partition, and evaluation episodes draw from a disjoint held-out partition. The harness samples initial states from that held-out partition under fixed seeds, so every method sees the same novel configurations in the same order. Changing the partition boundary, or letting seeds drift between methods, silently changes panel difficulty and invalidates the comparison.

A robust benchmark implementation starts with a manifest, not a model. The manifest says which episodes will run, which seeds instantiate them, which wrappers transform observations and actions, and which metric script produces the final table. The same manifest must evaluate the baseline and the proposed method.

1. Write a manifest with task panel, split, seeds, simulator build, wrappers, policy checkpoint, and metric function.
2. Run one deterministic smoke episode and verify that the saved trace matches the manifest.
3. Run every method through one evaluation script, with no method-specific termination or reward shaping.
4. Save per-seed metrics, aggregate metrics, videos or traces, and structured failure labels in one artifact.
5. Promote only same-config comparisons to the paper, keeping exploratory mismatches in diagnostics.

Code Fragment 2 shows the manifest shape that later sections reuse. A JSON file with this schema is more valuable than a screenshot because it makes the evaluation replayable.

# Build a replayable benchmark manifest before training or tuning.
# The manifest captures the evidence boundary that every compared method
# must share for the result to support a paper-facing claim.
from dataclasses import dataclass, asdict

@dataclass
class BenchmarkManifest:
    task_panel: str
    split: str
    seeds: tuple[int, ...]
    simulator: str
    wrappers: tuple[str, ...]
    metric: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

manifest = BenchmarkManifest(
    task_panel="pick-place-v1",
    split="heldout_objects",
    seeds=(0, 1, 2, 3, 4),
    simulator="ManiSkill3",
    wrappers=("rgbd_observation", "dense_action_normalization"),
    metric="success_rate",
)
print(manifest.as_row())

Expected output: the printed manifest should expose the task panel, split, seed policy, simulator, wrapper stack, and metric. If one of those fields is missing, the result is not yet an evaluation artifact.

A frozen manifest like the one above is also the first thing to consult when a comparison surprises you, because it lets you separate a measurement fault from a model fault. When a benchmark comparison fails, first ask whether the method failed or the measurement failed. Check for train/test overlap, seed tuning, wrapper drift, simulator-version drift, metric changes, and hidden data augmentation before assigning the result to model quality. This pattern turns a surprising leaderboard row into a reusable diagnostic asset.