Section 12.6: Reading a leaderboard without fooling yourself

"Every leaderboard row hides a configuration. Reading rank without reading provenance is reading a conclusion without the evidence."

A Skeptical Benchmark Reader

What This Section Builds

Swap two words in a config file you never see, and the same robot policy can jump twenty points on a leaderboard without learning anything new: this section gives you the checklist that catches that swap before it fools you into reading protocol drift as a method win.

It makes leaderboard reading operational, helping you decide whether a result compares methods or whether it accidentally compares different tasks, splits, seeds, simulators, wrappers, or tuning budgets. Figure 12.6B captures the whole flow: each row passes through a five-field provenance audit, and only rows that match on every field are promoted to a direct comparison.

The goal is not cynicism. The goal is disciplined trust: promote only construct-matched metrics that were co-computed in one pass on one configuration, and keep everything else in diagnostics.

Theory

A leaderboard row is a summary of many design choices. The visible number may be success rate, Success weighted by Path Length (SPL), reward, normalized score, completion rate, or throughput, but the hidden denominator is the episode panel and evaluation protocol. If two rows use different denominators, subtracting them does not estimate a method effect.

Why the panel is the hidden denominator

Mechanically, an episode panel is a fixed, enumerated set of initial conditions: object poses, scene geometry, lighting seeds, and task goal specifications drawn before evaluation begins. The panel is frozen so that every method faces identical starting states. When two rows use different panels, they computed success rates over different denominators, and subtracting them produces a number with no physical interpretation.

In embodied AI, the episode panel is not an abstract accounting choice. A robot that succeeds on panel A (short corridors, static obstacles, lab lighting) and fails on panel B (cluttered homes, moving people, variable illumination) has learned different physics. Changing the panel changes three things at once: the physical difficulty regime, the sensor noise profile, and the contact geometry (the shape and compliance of surfaces the robot's end-effector or body must physically touch) the policy must handle. A softer physical environment in the denominator can explain a leaderboard number that looks like a method improvement.

The practical design rule is one line: table comparisons need one evaluation script, one config, and one artifact that contains all compared rows. This is why the Habitat navigation challenges froze a fixed Gibson and Matterport3D episode set with a canonical SPL script, and why Meta-World pins its MT50 task list and reward wrappers. Suppose ETH Zurich's ANYmal locomotion results and a tabletop Franka manipulation policy both report "success." The numbers subtract cleanly only when the same Isaac Gym contact solver, the same camera resolution, and the same termination condition produced both. Cross-paper numbers, say an OpenVLA score quoted from one paper against an RT-2 score quoted from another, can motivate discussion, but they should not be written as direct wins unless the benchmark protocol is demonstrably the same.

A leaderboard number without a provenance key is not a result; it is a rumor with a decimal point.

Worked Example

Because that provenance key is the line between a result and a rumor, the first thing to build is code that computes it, grouping rows by exactly the fields that must match before a comparison is allowed.

Code Fragment 1 checks whether rows can share a paper table. It rejects a direct comparison when one method uses a different split, seed set, or metric, even if the row names come from the same benchmark family.

# Group leaderboard rows by the protocol fields that make them comparable.
# A method win is paper-facing only when all compared rows share
# panel, split, seed set, wrappers, simulator, and metric.
rows = [
    {"method": "baseline", "panel": "MT50", "split": "official", "seeds": (0, 1, 2), "metric": "success"},
    {"method": "candidate", "panel": "MT50", "split": "official", "seeds": (0, 1, 2), "metric": "success"},
    {"method": "ablation", "panel": "MT50", "split": "tuned_validation", "seeds": (0, 1, 2), "metric": "success"},
]

groups = {}
for row in rows:
    key = (row["panel"], row["split"], row["seeds"], row["metric"])
    groups.setdefault(key, []).append(row["method"])

print(groups)

Practical Recipe

1. Identify the construct: manipulation generalization, lifelong transfer, long-horizon household progress, navigation efficiency, social safety, or simulation throughput.
2. Check whether every compared row shares the same panel, split, seeds, wrappers, simulator build, and metric script.
3. Prefer seed-level values and confidence intervals over one best-run number.
4. Audit whether validation data, public test episodes, prompt templates, or generated-scene seeds influenced tuning.
5. Keep mismatched cross-paper numbers in a diagnostic note rather than a win table.

The algorithm below turns that recipe into a step-by-step audit you can run on any candidate row pair, producing a promote-or-diagnostic verdict and a manifest of which provenance fields matched.

Reading a leaderboard well means converting a ranked list into an evidence map. A ManiSkill3 speed result, a Meta-World success result, a Habitat SPL result, and an Isaac Lab throughput result can all be valuable, but they answer different questions. The audit is to keep each number attached to the question it actually answers.

The graduate-level habit is to distinguish result validity from comparison validity. A row can be valid for its own protocol and still invalid as a direct comparison to another row. A paper-facing claim needs both: the row must be internally reproducible, and the compared rows must be co-computed on one configuration. To see how large the gap can get: one team running five seeds on the official split reports 61%; another team reporting the best of fifteen seeds on a validation-tuned split reports 74% on the same benchmark name. Selecting the best of fifteen seeds instead of averaging five is like picking the tallest student out of a class of thirty and reporting that as the average height of the class. The 13-point difference is entirely protocol, not method.

Before scanning the checklist below, guess: how many of the five protocol fields can you name from memory for the last leaderboard row you read?

Answering that checklist consistently is only possible if each row carries its provenance with it, which is why the next step is to fix the storage format rather than the reading habit alone. A robust leaderboard entry starts as a machine-readable row with provenance fields. Store the model checkpoint, benchmark version, split, seeds, wrappers, simulator, metric script, hardware when throughput is measured, and the raw per-episode outputs. A reviewer should be able to reconstruct the table from that row.

1. Build a provenance table before copying any result into prose.
2. Group rows by task panel, split, seeds, wrappers, simulator, and metric.
3. Promote only groups with two or more methods to direct comparison.
4. Attach uncertainty or seed-level values to every promoted comparison.
5. Write mismatched rows as diagnostic context, not as wins.

Code Fragment 2 shows a row format that supports direct-comparison filtering. It is deliberately boring, because boring provenance is what keeps benchmark claims honest.

# Store leaderboard provenance with the metric value.
# The comparison key is everything except method and value because
# those are the fields a method is allowed to change.
from dataclasses import dataclass, asdict

@dataclass
class LeaderboardRow:
    method: str
    panel: str
    split: str
    seeds: tuple[int, ...]
    simulator: str
    metric: str
    value: float

    def as_row(self) -> dict[str, object]:
        return asdict(self)

row = LeaderboardRow(
    method="candidate",
    panel="Habitat-SocialNav",
    split="unseen_scenes",
    seeds=(0, 1, 2, 3, 4),
    simulator="Habitat 3.0",
    metric="success_weighted_by_path_length",
    value=0.51,
)
print(row.as_row())

Expected output: the printed row should expose the provenance fields needed to reconstruct the comparison. If a leaderboard omits these fields, treat it as scouting evidence until the configuration can be verified.

When a leaderboard comparison looks surprising, audit the denominator before the method. Rule out hidden split changes, seed tuning, wrapper drift, simulator drift, metric changes, throughput-hardware changes, and selective reporting first. Only what survives that audit needs a scientific explanation.