Section 9.3: Fidelity: physical, visual, behavioral

"Realism is not a volume knob. It is a mixing board with labels you should read before touching anything."

A Calibration-Minded AI Agent

Fidelity Is Axis-Specific

Ask an engineer whether their simulator is "realistic" and you will get a confident yes, yet the same simulator can be flawless for one task and useless for the next, because realism is not one property but several that fail independently. Physical fidelity concerns dynamics: mass, inertia, friction, contacts, compliance, and actuation. Visual fidelity concerns rendered images, lighting, materials, camera models, and occlusions. Behavioral fidelity concerns whether the environment responds in ways that matter for the task: doors open, objects move, receptacles contain, fluids pour, and failure states persist. Figure 9.3A places these concerns on a spectrum from coarse rigid-body physics to full photorealistic rendering, annotating each level with the engine, solver, and policy type that benefits most. Because a mismatch on any single axis can collapse an otherwise well-trained policy, practitioners must treat this as four independent transfer risks, not one. Figure 9.3B shows these four axes feeding a separate gap risk into policy transfer, so a failure on any one can sink the result even when the other three match reality.

A visually simple MuJoCo model can be enough for torque-control research. A perception policy may need Omniverse Replicator or BlenderProc imagery with camera artifacts. A household agent may need Habitat, AI2-THOR, ProcTHOR, BEHAVIOR, or OmniGibson-style object semantics because the claim depends on scene interaction, not just contact physics.

Worked Miniature: A Fidelity Match

Code Fragment 9.3.1 is a simple fidelity checklist. It maps task needs to simulator capabilities so the team can defend why a tool is sufficient for a particular claim.

# Match simulator capabilities to the task's transfer risks.
# The output exposes unsupported fidelity axes before training begins.
task_needs = {"contact", "depth_noise", "object_state"}
simulators = {
    "MuJoCo": {"contact", "actuation"},
    "Isaac Lab": {"contact", "depth_noise", "camera_rendering"},
    "ProcTHOR": {"object_state", "layout_diversity", "camera_rendering"},
}

for name, capabilities in simulators.items():
    missing = sorted(task_needs - capabilities)
    status = "ready" if not missing else f"missing {missing}"
    print(name, status)

MuJoCo missing ['depth_noise', 'object_state']
Isaac Lab missing ['object_state']
ProcTHOR missing ['contact', 'depth_noise']

Expected output: the trace identifies which simulator capabilities are missing for the task contract. That missing list is not a rejection of the simulator. It is the list of reality-gap assumptions that the experiment must either measure, randomize, or exclude from the claim.

Of the four axes the checklist tracks, behavioral fidelity is the easiest to overlook, because its failures show up only after many steps rather than on the first contact. Behavioral fidelity matters because long-horizon tasks require the world to remember what the agent did. A robot that opens a drawer, removes an object, and closes the drawer again needs the simulator to persist all three state changes independently. If the simulator resets object positions on each physics step, the policy learns to act in a world that never accumulates consequences. That skill does not transfer. In practice, teams running a five-step kitchen task without persistent object state have reported needing on the order of tens of thousands of episodes to stumble into a successful sequence, while the same policy with full state persistence has converged in well under a tenth of that; exact ratios vary by task and reward shaping, but the qualitative gap is consistent, because every partial success survives as a stepping stone rather than vanishing. On a real robot, a misplaced lid stays misplaced; a simulator with weak behavioral fidelity silently undoes the error, training the policy to ignore irreversible actions.

The checklist in Code Fragment 9.3.1 tells you a simulator is missing an axis; it does not tell you what to do about a missing behavioral axis specifically. In practice a team facing that gap has three options: switch to a simulator with native scene-state support (Habitat, BEHAVIOR, or OmniGibson), author a light-weight state-transition rule layer on top of an existing physics-only simulator (as described next), or narrow the claim to exclude any statement about persistent object state. The scene-state graph described below is the mechanism, not just a definition of the axis.

Simulators implement behavioral fidelity through a scene-state graph. Every interactable object carries a discrete state (open, closed, filled, broken) alongside its continuous pose. On each environment step, the physics engine updates pose. A separate rule layer then evaluates contact events and transitions object states. A drawer moves from closed to open when the end-effector applies force along the slide axis above the joint limit. That state transition holds until the agent applies the reverse action. This persistence is what makes consequences accumulate.

Choosing The Necessary Fidelity

Each task has its own transfer-critical mismatches. A legged locomotion policy hinges on ground contact, actuator delay, IMU (inertial measurement unit, the onboard sensor that reports orientation and acceleration) noise, terrain variation, and controller frequency. A vision-based grasp detector hinges on depth holes, reflective materials, occlusions, camera calibration, and object shape. A household agent hinges on object state and task semantics, where photorealistic walls barely register.

1. Write the decision the agent must make.
2. Name the simulator mismatch that could change that decision.
3. Choose the fidelity axis that controls the mismatch.
4. Measure the mismatch directly, or record why it is outside the current claim.
5. Keep the claim as narrow as the supported fidelity.

Fidelity: physical, visual, behavioral becomes useful when it is tied to a closed-loop contract. In this chapter on Why Simulation Is Central, the contract names the observation stream, the state estimate, the action representation, the timing budget, and the evaluation artifact. Without that contract, a model can look capable in a notebook while failing the first time a sensor drops a frame or a controller saturates.

That contract also forces a discipline on how results are reported, because each part of it backs a different kind of claim. For fidelity, separate the conceptual claim, the systems claim, and the evidence claim. "Soft contacts should help deformable grasping" is a mechanism; "MuJoCo's MJX soft-body solver runs at 5,000 envs on an A100" is a systems claim; "the policy lifts a silicone cup 87% of the time on a real Franka" (the Franka Emika Panda, a widely used 7-degree-of-freedom research arm) is a closed-loop result. Keep their evidence separate: a working MJX solver does not prove the grasp transfers, and a high sim success rate does not prove the contact model matches the real silicone. Reporting one as if it backed another is how a 94%-in-Isaac-Lab number becomes a 40%-on-hardware surprise.

Start with a small baseline that logs inputs, outputs, units, timestamps, and termination conditions before moving to Gymnasium or PettingZoo. The library run should keep the same artifact schema, so the comparison remains a same-task evaluation.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

Before reading on, guess: a locomotion policy achieves stable trotting in simulation but falls within three steps on the physical robot. Which single fidelity axis is most likely responsible? Hold your answer while you read the diagnostic below.

When an experiment about fidelity: physical, visual, behavioral fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

Consider a specific case. A locomotion policy trained in MuJoCo at 500 Hz achieves stable trotting in simulation, but falls within three steps on the real Unitree A1 robot (a small quadruped research platform). Step one logs the real robot's joint torques and compares them against the simulated torques at the same gait phase. If the real torques run consistently 15 to 20 percent higher, the gap points to actuator-model fidelity: the simulation used an idealized torque-controlled joint, while the real hardware applies PD control (a proportional-derivative feedback controller that sets torque from position and velocity error) at 1 kHz with its own latency. Step two introduces a simulated actuator delay of 10 ms and reruns training. Step three re-evaluates on the physical robot. If the success rate recovers from near-zero to above 80 percent under the same conditions, the diagnosis holds, and the experiment now has a falsifiable fidelity fix rather than a narrative about the "reality gap."