Section 52.5: Reproducible evaluation: SIMPLER and sim-as-proxy

"A benchmark you cannot rerun is an anecdote with a decimal point."

An Evaluation Methodologist

Why This Matters

Two policies can post identical average success rates in the simulator and still send opposite recommendations to the deployment team, because the simulator ranked them in the wrong order and nobody checked. Evaluation choices rewrite the scientific claim: if the metric drops time, energy, or safety terms that the deployment team cares about, the benchmark no longer matches the real decision. Figure 52.5.1 frames this as a sim-to-real evaluation ladder: a simulator panel is a useful proxy only when its limits are named rather than ignored.

Let \(S\) be the simulator score and \(R\) the real-world score over matched tasks. A practical proxy check is the fidelity gap $$\Delta = \left|\mathbb{E}[S] - \mathbb{E}[R]\right|$$ plus a rank correlation between methods over the same task panel. A low average gap without rank preservation is still a weak proxy.

The fidelity gap matters in embodied AI because physical consequences scale with systematic bias. A simulator that consistently inflates grasp success by 15 percentage points pushes teams to ship policies that fail on real hardware. Those failures carry costs ranging from dropped objects to joint torque faults that require a manual reset. A small \(\Delta\) does not mean the proxy is safe. It means the absolute scores are close, which is a different and weaker claim than trusting the ranking. Figure 52.5.2 lays out the two-check pipeline that separates these claims: the fidelity gap and the rank correlation are computed independently, and proxy validity requires both to pass. The rank correlation check in the figure uses the Spearman coefficient (rho, a value between -1 and 1 measuring how consistently two rankings agree, walked through step by step later in this section); for now it is enough to know that rho close to 1 means the simulator and the real panel agree on which policy is best, and rho near 0 means they do not.

A policy that works in simulation but fails on hardware is not a policy: it is a wishlist backed by a flawed mirror.

To estimate \(\Delta\), run the same evaluation episodes in both domains. \(\mathbb{E}[S]\) is the mean success rate across episodes and seeds in the simulator. \(\mathbb{E}[R]\) is the mean over a matched real panel that uses the same task definitions, object instances, and success criteria. Matched means the episode start states, object poses, and scoring thresholds align across both domains. Any remaining difference then reflects the simulator, not protocol drift.

Worked Example

A simulator may preserve policy ranking on tabletop grasping but fail to preserve contact-rich insertion errors because friction and compliance are mis-modeled. That makes it a good filter for broad candidate screening but a weak final judge for insertion policies.

sim_scores = {"A": 0.84, "B": 0.76, "C": 0.72}
real_scores = {"A": 0.68, "B": 0.70, "C": 0.61}
ranking_sim = sorted(sim_scores, key=sim_scores.get, reverse=True)
ranking_real = sorted(real_scores, key=real_scores.get, reverse=True)
mean_gap = round(sum(abs(sim_scores[k] - real_scores[k]) for k in sim_scores) / len(sim_scores), 3)
print({"ranking_sim": ranking_sim, "ranking_real": ranking_real, "mean_gap": mean_gap})

{'ranking_sim': ['A', 'B', 'C'], 'ranking_real': ['B', 'A', 'C'], 'mean_gap': 0.11}

Expected output: The proxy loses trust here because the best simulator method is not the best real-world method. The mean gap alone would miss that decision-level failure.

With those tools in place to enforce matched schemas and align episodes, the payoff of a disciplined correlation audit becomes concrete in a published benchmark. Consider a specific case: the SIMPLER benchmark (Li et al., 2024) evaluates manipulation policies inside a MuJoCo-based tabletop environment (MuJoCo is the rigid-body physics engine introduced by Todorov et al., 2012) that mirrors the Google Robot and Bridge v2 real-robot setups, where Bridge v2 is a widely used public dataset of teleoperated WidowX manipulation episodes, used in the RT-2 and Octo evaluations, where RT-2 is a vision-language-action model from Google DeepMind and Octo is an open-source generalist transformer policy from UC Berkeley.

On pick-and-place tasks with static objects, SIMPLER achieved Spearman rank correlations above 0.9 with physical robot scores across five candidate policies. On tasks requiring fine contact alignment (e.g., peg insertion), the same simulator dropped to correlations near 0.5, making it useful for initial screening but unreliable for final selection. That explicit per-task-family validity table is what distinguishes a credible sim-as-proxy claim from an unchecked one. The scale difference is striking: a full five-policy comparison on the physical robot required roughly 1,200 real episodes, while the simulator screened the same candidates in 80 episodes worth of real-robot time, cutting physical evaluation cost by more than 90 percent before the correlation audit even ran.

That open-source workflow worked precisely because its trust was scoped task by task, which points to the general rule governing every credible proxy claim.

Scoping the proxy claim

The strongest simulator proxy claims are narrow and explicit. A simulator might be trusted for policy ranking within one morphology and camera setup, but not for fleet-wide energy forecasting or human-interaction safety.

The practical deliverable is a proxy-validity table that names the task family, robot configuration, simulator settings, real-world panel, correlation window, and known mismatch. Without that table, a simulator result is evidence for simulation only.

A common failure mode is to treat a proxy as universally valid after one early correlation result. Proxy validity is local to task family, hardware configuration, and decision type.

Cross-References

This section connects backward to Chapter 20 on sim-to-real transfer and forward to Section 52.6 on benchmark hygiene.