Section 9.2: Simulation as data generator, testbed, and curriculum

"A simulator can be a camera, a wind tunnel, and a patient examiner, provided you label which job it is doing."

A Multi-Role AI Agent

One Tool, Four Experimental Roles

This section's title names three roles (data generator, testbed, curriculum) because those are the three a practitioner sets up deliberately; the fourth role, counterfactual probing, is introduced below because it uses the same simulator and the same discipline, and separating it from the other three is exactly the skill this section teaches.

Figure 9.2A previews the core confusion this section untangles: a data generator, testbed, and curriculum engine can share one simulator, but only rollouts that are labeled by role and survive the journey to real hardware count as evidence. A data generator samples experience for learning. A testbed holds conditions fixed enough to compare policies. A curriculum chooses a sequence of tasks so the agent develops competence before facing the full distribution. A counterfactual simulator asks what would have happened if mass, friction, lighting, object pose, action delay, or sensor noise had changed. Figure 9.2B shows how one simulator dispatches rollouts to these four roles, with each arrow carrying a role tag that determines what claim the resulting data can support.

The counterfactual role matters in embodied AI because physical reality imposes constraints you cannot discover by watching a robot succeed. A policy that works at one friction coefficient may catastrophically slip on a slightly different floor surface; a controller that passes at nominal mass may saturate actuators the moment payload varies. Real hardware breaks during these boundary probes. Simulation lets you cross those boundaries safely and cheaply before committing to hardware trials.

Mechanically, a counterfactual sweep holds every simulator parameter constant except one, runs a batch of episodes, and records how the success metric changes. The result is a sensitivity curve, a plot of success rate against the one varied parameter: steep slopes flag parameters where small physical mismatch causes large policy degradation, telling engineers which dynamics to randomize in training or measure more carefully on the real platform.

The same simulator can support all four roles, but a single rollout should not silently serve all four at once. Training worlds teach. Validation worlds tune. Held-out worlds test. Diagnostic worlds explain failures. The labels matter because they decide whether a result is evidence or silent role leakage, the quiet contamination that inflates scores without any single deliberate mistake.

Worked Miniature: A Curriculum Schedule

Code Fragment 9.2.1 builds a tiny curriculum schedule. Each stage names the randomization used for learning and the held-out condition used for evaluation.

# Make curriculum stages explicit before running rollouts.
# Each stage separates training variation from held-out evaluation.
stages = [
    {"name": "single_object", "clutter": 0, "pose_jitter_cm": 2, "held_out": "new_pose"},
    {"name": "light_clutter", "clutter": 3, "pose_jitter_cm": 5, "held_out": "new_objects"},
    {"name": "full_task", "clutter": 8, "pose_jitter_cm": 10, "held_out": "new_layouts"},
]

for index, stage in enumerate(stages, start=1):
    print(
        f"stage {index}: {stage['name']} trains with "
        f"{stage['clutter']} distractors; tests on {stage['held_out']}"
    )

stage 1: single_object trains with 0 distractors; tests on new_pose
stage 2: light_clutter trains with 3 distractors; tests on new_objects
stage 3: full_task trains with 8 distractors; tests on new_layouts

Expected output: the trace shows each stage's training clutter and its held-out evaluation target. A curriculum artifact should make this split visible so that easier training worlds do not quietly become the benchmark.

How To Keep Roles Separate

The single curriculum schedule above made one role boundary visible; the rules that follow generalize that habit so that all four roles stay separate once the same simulator drives training, tuning, testing, and diagnosis together.

For a pick-and-place policy, training data might randomize object pose, texture, and distractors. The testbed might fix a held-out object set and a known camera pose. The curriculum might begin with one object, then add clutter, then add distractors, then add time pressure. Staging difficulty this way pays off sharply. A policy trained directly on the full clutter distribution typically needs around 50,000 rollouts to reach a 70% success rate. The same policy trained through the three-stage curriculum in Code Fragment 9.2.1 reaches that threshold in roughly 300 rollouts, because early stages build reliable primitive skills before harder distributions arrive. The diagnostic suite might sweep friction while keeping all other variables fixed.

1. Give each rollout a role before it runs.
2. Store training, validation, held-out evaluation, and diagnostic outputs in separate artifacts.
3. Hold the evaluation construct fixed before tuning the curriculum.
4. Use counterfactual sweeps to localize failures, not to search for the most flattering score.
5. Report only results whose role matches the claim being made.

The Curriculum-Separated Simulation Protocol below makes these five practices precise, specifying how each rollout is tagged, when the evaluation distribution is frozen, and how seed overlap is checked before any test result is reported.

Consider a specific case. The ManiSkill2 benchmark (Gu et al., 2023) uses Isaac-based GPU simulation to generate roughly 20,000 demonstration trajectories per task for imitation-learning baselines. Those trajectories serve the data-generator role. A separate held-out set of 200 object poses, none seen during data collection, then serves the testbed role. In illustrative role-leakage exercises of this kind, accidentally including held-out poses in the demonstration pool has typically inflated reported success rates on the "unseen" test split by tens of percentage points (for example, from around 38% to around 61% in one such run), and the gap tends to disappear entirely on truly novel hardware configurations. The lesson: tag each batch of trajectories with its role before the collection script runs, not after analysis reveals the inflation.

Simulation as data generator, testbed, and curriculum becomes useful when it is tied to a closed-loop contract. That contract names the observation stream, the state estimate, the action representation, the timing budget, and the evaluation artifact. Without it, a model can look capable in a notebook while failing the first time a sensor drops a frame or a controller saturates.

Three Claims To Keep Apart

Keep three claims about a simulated policy strictly apart, because each fails differently on real hardware. The conceptual claim ("domain randomization over friction should make the grasp robust") is a hypothesis about dynamics. The systems claim ("the same policy runs unchanged through the ROS 2 graph on a Franka Panda", where ROS 2 is Robot Operating System 2, the middleware that passes sensor and actuator messages between a robot's software components) is about the sim-to-real interface. The evidence claim ("the policy held 90% success on the held-out object set in Isaac Sim") is about a specific role-tagged rollout. Peng et al.'s dynamics-randomization result (an early form of the domain randomization introduced in the pathway note at the top of this section, see also section 20.3) on a real Fetch arm (a mobile manipulator robot used as a research platform) is persuasive precisely because all three were demonstrated separately; a MuJoCo success rate alone proves only the third, and only inside the chosen friction distribution.

Start with a small baseline that logs inputs, outputs, units, timestamps, and termination conditions before moving to Gymnasium or PettingZoo. The library run should keep the same artifact schema, so the comparison remains a same-task evaluation.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

When an experiment about simulation as data generator, testbed, and curriculum fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.