Section 1.4: Physical vs. simulated embodiment

"Simulation gives you replay. Hardware tells you which assumptions survived contact."

Section 1.4

Two worlds, one policy

A walking policy can clear ten thousand simulated obstacles flawlessly and then snap a real robot's ankle on the first physical step, and the entire reason fits in one equation. Treat the simulator and the robot as two transition processes that share an action space. The simulator advances state by a declared model \(T_{\text{sim}}(s' \mid s, a)\): equations the modeler wrote down for rigid-body dynamics, a contact solver, a friction law, and a sensor model. The physical robot advances by \(T_{\text{real}}(s' \mid s, a)\), which no one ever wrote down in full. The same policy \(\pi\) induces a trajectory distribution in each. It therefore yields two values of the task functional \(J(\pi)\) from Section 1.1. Figure 1.4A illustrates this with a real robot accumulating wear and latency beside its faster-than-real-time simulated twin. The reality gap is their difference,

$$\Delta(\pi) = J_{\text{sim}}(\pi) - J_{\text{real}}(\pi).$$

This is the central object of the section. It is positive when a policy looks better in simulation than on hardware, which is the usual case, and it is measured in the units of \(J\) (success rate, return, completion time), not asserted as a caveat. A sim-to-real method is good exactly to the extent that it makes \(\Delta(\pi)\) small for the policies you actually deploy, and the honest way to report transfer is to give \(J_{\text{sim}}\), \(J_{\text{real}}\), and \(\Delta\) together, computed on the same policy and the same task definition.

Before reading on, ask yourself: if a policy scores 94% in simulation, what is your best guess for its success rate on the physical robot? Keep that number in mind as you work through the rest of this section.

A policy that succeeds only in simulation has not solved the task; it has solved the simulator.

What simulation buys, and at what fidelity

Simulation is attractive for reasons that are concrete and quantifiable. It runs faster than wall-clock time, often by orders of magnitude, and it runs thousands of environments in parallel on a GPU, so a learning run that would take physical years finishes in hours. A locomotion policy that requires roughly 50,000 physical rollouts to converge when trained on a single real robot typically needs only 300 rollouts worth of hardware time when the same compute runs in 4,096 parallel simulated environments; the robot is no longer the bottleneck, the policy is. It offers perfect resets: every episode can start from an identical state, which makes counterfactual experiments (change one variable, hold the rest) possible in a way hardware never allows. It exposes ground-truth state (exact poses, contact forces, object masses) that on a robot must be estimated and is partly unobservable. And its failures are free: a simulated robot can fall ten thousand times with no broken gearbox and no safety incident, because simulation offers physics without consequences.

What it does not give for free is agreement with the physical world. The divergence has named sources, and they map onto three core fidelity dimensions worth keeping separate (a fourth, timing, is named just below):

• Physical fidelity covers dynamics: contact and friction are stiff, discontinuous, and the hardest thing for any solver to get right; restitution (how much bounce a collision retains), compliance, joint backlash (the small dead-zone play in a gear train before it engages), motor torque saturation, and actuation latency are routinely simplified or omitted.
• Visual fidelity covers the appearance gap that a vision policy sees: lighting, materials, textures, reflections, and the difference between a rendered image and a camera frame with real noise, motion blur, and rolling shutter.
• Behavioral fidelity covers whether the agent makes the same decisions and the same mistakes in both worlds. This is the only dimension that ultimately matters, and the one that physical and visual fidelity are merely means toward. A scene can look photorealistic and still produce the wrong failure.

A fourth dimension, temporal fidelity (whether the simulator's control timing, including sensor and actuation delay, matches the robot's), is named here for completeness because the comparison table later in this section scores all four dimensions; it gets its own callout right after that table appears.

These trade against speed. A high-fidelity contact solver with fine time steps and a path-traced renderer is slower per step, which shrinks the parallelism that made simulation attractive in the first place. The speed-fidelity tradeoff is the practitioner's core dial: you buy fidelity with compute, and past some point the marginal fidelity does not change which policy you would deploy, so paying for it is waste.

Closing the gap: identification and randomization

Once you accept that fidelity is only ever "sufficient for claim X," the practical question becomes how to shrink the gap that remains. Two strategies dominate, and they pull in opposite directions. System identification narrows the simulator onto the specific robot: measure masses, friction coefficients, motor constants, and latencies, then fit \(T_{\text{sim}}\) to match logged hardware rollouts. Fitting the model accurately shrinks \(\Delta\). The limit is scope: you can only identify parameters you thought to measure, and residual unmodeled physics bites on deployment. Domain randomization takes the opposite bet. Rather than match one robot, it samples uncertain parameters (friction, mass, latency, lighting, textures) over a wide range during training. The policy must be robust to all of them. It then treats the real world as just one more sample from the training distribution. This trades peak in-distribution performance for transfer that requires no accurate model. In practice the two strategies combine: identify what you can measure, randomize over what you cannot. Chapter 13 develops simulation and the physics engines; Chapters 20 and 43 develop domain randomization and sim-to-real transfer in depth.

A reality gap you can run

The checklist above stays abstract until you watch \(\Delta\) appear in a system small enough to hold in your head, so here is one. The code below is the smallest honest demonstration of \(\Delta\). A 1D braking controller must stop a sliding mass before a wall, planning its braking distance from an assumed friction coefficient (the value it was tuned on, \(\mu = 0.8\)). We then run the identical controller, no retuning, on hardware-like surfaces where the true friction is lower. Correct in its own world and crashing in the other, it lets us read off the reality gap as the drop in success rate.

# Reality gap for a 1D braking controller tuned to one friction coefficient.
# Same policy, two worlds: it succeeds where mu matches and overshoots where mu drops.
import numpy as np

def overshoot(mu_true, mu_assumed, v0=4.0, dt=0.005, g=9.81, margin=0.30):
    # The controller knows the wall position and plans its braking distance from
    # its ASSUMED friction. It must stop on or before the wall (overshoot <= 0).
    planned_stop = v0**2 / (2.0 * mu_assumed * g)
    wall = planned_stop + margin
    brake_point = wall - planned_stop - v0 * dt   # one-step lead removes integration bias
    x, v = 0.0, v0
    while v > 1e-4:
        decel = mu_true * g if x >= brake_point else 0.0
        v = max(0.0, v - decel * dt)
        x += v * dt
    return x - wall                               # > 0 means it crossed the wall (a crash)

mu_nominal = 0.8                                   # the value the controller was tuned on
print(f"{'true mu':>8} {'overshoot (m)':>14} {'outcome':>8}")
for mu_true in (0.8, 0.6, 0.4, 0.3):
    ov = overshoot(mu_true, mu_assumed=mu_nominal)
    print(f"{mu_true:>8.2f} {ov:>14.3f} {'ok' if ov <= 0 else 'CRASH':>8}")

# Reality gap as a scalar: success at the tuned value vs. over a realistic friction band.
succeeds = lambda mu: overshoot(mu, mu_nominal) <= 0
J_sim  = float(np.mean([succeeds(0.8) for _ in range(50)]))                 # eval where it was tuned
J_real = float(np.mean([succeeds(mu) for mu in np.linspace(0.3, 0.8, 50)])) # eval over the real band
print(f"\nJ_sim  (mu fixed at 0.80)  = {J_sim:.2f}")
print(f"J_real (mu in [0.30,0.80]) = {J_real:.2f}")
print(f"reality gap  Delta         = {J_sim - J_real:.2f}")

Which fidelity dimension transfers what

The table separates the four fidelity dimensions (the three named above plus temporal) and states, for each, what a simulator typically gets right, where the gap usually opens, and which gap-closing strategy applies. It is a planning aid: locate the claim you want to make in the left column and read across.