Section 13.6: Randomization vs. realism; measuring transfer readiness

"Randomization hedges against gaps you cannot measure. Realism closes gaps you can. Measure first, then choose your weapon."

A Careful Control Loop

What This Section Builds

Two teams ship the same simulator score of 0.92. One robot grasps every object on the line; the other drops the first part it touches. The difference was never in the score. It was in whether anyone measured the gap between that score and reality before the hardware arrived. This section makes that measurement operational, explaining how to decide when to spend effort on broader randomization, better realism, or a hybrid that reconstructs measured factors and randomizes residual uncertainty.

The goal is a comparison that a skeptical reader can audit number by number: one task panel, one configuration, one seed policy, one metric definition, and one artifact containing all compared results.

Theory

A useful readiness score starts with construct matching. If the claim is about grasp success, compare grasp success on the same object panel, not detector average precision from one run and robot success from another. If the claim is about pose robustness, compare pose error under the same camera split, not a different render split.

Randomization is preferable when the real distribution is broad, uncertain, and too expensive to reconstruct precisely. Realism is preferable when measured details dominate failure, such as camera calibration, object scale, or contact geometry. A hybrid is often strongest: reconstruct what can be measured, then randomize residual uncertainty around it. The scale of this distinction is large. A baseline policy trained under a single fixed friction value may need 40,000 episodes to reach 70 percent real success. A hybrid reconstructs the measured friction range and randomizes only residual mass uncertainty, and it reaches the same threshold in roughly 800 episodes. The training distribution no longer wastes capacity on physically impossible conditions. Figure 13.6B lays out this decision process as a flow from failure log to strategy choice to shared evaluation. Concretely, a strategy is transfer-ready once its gap \(\Delta(s)\) is documented and small enough for the deployment step and its gain \(g(s)\) over baseline is the largest among the strategies compared; the algorithm below formalizes both terms.

Worked Example

That identical evaluation path is easiest to see in code, where forcing every strategy through one shared data structure is exactly what makes the comparison auditable.

The following snippet computes a simple transfer-readiness comparison from one shared panel. The important detail is that randomization, realism, and hybrid scores are stored together rather than copied from separate experiments.

# Compare transfer strategies on one shared evaluation panel.
# Every number uses the same metric, scenes, seeds, and failure labels.
results = {
    "baseline": {"real_success": 0.58, "slip_failures": 21},
    "randomized": {"real_success": 0.71, "slip_failures": 12},
    "realistic": {"real_success": 0.67, "slip_failures": 15},
    "hybrid": {"real_success": 0.76, "slip_failures": 8},
}

for method, metrics in results.items():
    gain = metrics["real_success"] - results["baseline"]["real_success"]
    print(f"{method}: success={metrics['real_success']:.2f}, gain={gain:.2f}")

Practical Recipe

1. Choose the real or proxy panel that will define transfer readiness before training variants.
2. Use the same success metric, failure taxonomy, scene split, and seed policy for every method.
3. Compare baseline, randomization, realism, and hybrid strategies in one evaluation artifact.
4. Report real success, major failure labels, and the gap between simulator and real performance.
5. Treat any number from a different configuration as diagnostic context, not as part of the main comparison.

Once that audit checklist is in hand, the stakes of skipping it come into focus.

A policy that works in simulation but collapses on hardware is not a policy: it is a rehearsal that never met its audience.

Randomization, realism, and hybrid strategies become useful when they are judged by the same closed-loop evidence. The artifact should include the simulator score, real score, the measured sim-to-real gap, failure labels, and exact split identifiers.

Separate the three claims

The graduate-level habit is to separate three claims. The simulator claim says the method performs under synthetic conditions. The transfer claim says it performs on held-out real or proxy conditions, which requires measuring transfer performance on a matched evaluation panel. The readiness claim says the gap and failure labels are small enough for the next deployment step.

A robust implementation starts with a single comparison artifact. Extend the results schema from Code Fragment 1 so every method entry records simulator score, real score, gap, metric, and split under one identical schema.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

What a complete artifact exposes: for every method the stored record should expose method name, simulator score, real score, metric, split, and sim-to-real gap. If one of those fields is missing, the record is not yet an evaluation artifact.

When a transfer-readiness comparison fails, first check whether the metrics are construct-matched and co-computed. Then inspect the failure labels: a large sim-to-real gap with many perception failures calls for rendering or sensor work, while a small gap with many contact failures points to dynamics, controller, or task coverage.