Section 20.1: The reality gap revisited

"The MDP you trained on and the world you deployed into are not the same problem. The gap between them is not a bug in the simulator; it is a bill the real world sends at inference time."

Section 20.1

Two centimeters: that is how far a flawless ten-million-step grasping policy can miss on its very first real-world reach, and the maddening part is that the simulator was never wrong, just answering a different question. This section takes that single failure label, "the reality gap," and decomposes it into six measurable mismatch types that each name a specific interface to test and repair. As Figure 20.1A shows, the gap is never one crack in the simulator: it is the combined effect of perception, dynamics, timing, contact, and evaluation mismatches surfacing together in the same closed loop.

By the end of this section you will be able to do three concrete things: name each of the six gap types by the interface it corresponds to, run the paired-rollout diagnostic algorithm below to localize which gap dominates a given failure, and choose a narrow repair (recalibrate a parameter, add a delay buffer, adjust an observation model) instead of defaulting to broad fixes such as full domain randomization or higher simulator fidelity.

This section turns the reality gap from a slogan into a set of measurable mismatches. The most common gaps are observation mismatch, transition mismatch, actuator mismatch, timing mismatch, contact mismatch, and evaluation mismatch. Each one creates a different debugging question. A policy trained for ten million simulator steps can fail its first real-world rollout because one interface changed enough to flip the action ranking, not because the entire model is wrong. Without knowing which interface diverged, teams routinely burn 50,000 additional rollouts chasing the gap; with a paired trace that names the culprit in the first diagnostic run, the same repair often takes fewer than 300.

The key question is practical: when a policy succeeds in simulation and fails on hardware, what engineers face is the sim-to-real gap, and the productive response is not to distrust the simulator but to identify which specific interface changed enough to invalidate the learned action. In practice, a friction coefficient shift from 0.6 in simulation to 0.35 on a real surface can be enough, for contact-sensitive tasks such as peg insertion, to drop task success from roughly 95% to under 30%, even while every other simulator parameter remains identical.

A policy that performs flawlessly in simulation but collapses on the first hardware rollout is not a failed policy; it is an undiagnosed one.

Theory

A useful formalization compares the simulator transition model \(P_{\text{sim}}(s_{t+1}\mid s_t,a_t)\) with the hardware transition model \(P_{\text{real}}(s_{t+1}\mid s_t,a_t)\). The policy never sees these distributions directly. It experiences them as different next observations, different rewards, and different safety margins after the same nominal action.

The gap is load-bearing (meaning it is the cause actually holding up the failure, not an incidental difference) when it changes the policy ranking: action \(a_1\) looks better than action \(a_2\) in simulation, but the ordering reverses on the robot. Small parameter errors matter most when they push the policy across a contact threshold, actuator limit, sensor blind spot, or termination condition.

The isolation step below relies on the policy gradient \(\nabla_\theta J\), the direction in which the policy's parameters \(\theta\) would move to increase expected return \(J\) under the current transition model; a sign flip in that gradient for a perturbed parameter is the formal signal that the parameter, not noise, caused the ranking reversal described above.

Worked Example

Code Fragment 20.1.1 below shows a tiny diagnostic for a pushing policy. It compares the same commanded push in simulation and on hardware, then labels the dominant gap instead of hiding the failure behind one success rate.

# Compare a simulated push with a hardware push using the same command.
# The gap label points to the interface that changed the rollout outcome.
sim_trace = {"slip_cm": 0.4, "settle_ms": 110, "success": True}
real_trace = {"slip_cm": 2.1, "settle_ms": 190, "success": False}

if real_trace["slip_cm"] - sim_trace["slip_cm"] > 1.0:
    gap = "contact and friction"
elif real_trace["settle_ms"] - sim_trace["settle_ms"] > 50:
    gap = "actuator delay"
else:
    gap = "evaluation or observation"

print(f"sim_success={sim_trace['success']}, real_success={real_trace['success']}")
print(f"dominant_gap={gap}")

Expected output: a useful reality-gap diagnostic reports the simulator outcome, the hardware outcome, and the suspected mismatch category. If the trace contains only final reward, the team cannot tell whether to fix sensing, dynamics, actuation, timing, or evaluation.

Practical Recipe

1. Write the simulated MDP assumptions: state variables, observation noise, transition parameters, actuator model, contact model, and termination rule.
2. Record the hardware interface with the same fields: sensor timestamps, command timestamps, controller status, measured motion, safety events, and success label.
3. Run paired rollouts with the same initial condition family and the same commanded policy checkpoint.
4. Label failures by the first interface that diverges enough to change the action outcome.
5. Repair the narrowest mismatch first, then rerun the paired diagnostic before changing the policy architecture.

Naming the likely gap for a task is only half the discipline; the other half is committing to a contract that makes any two rollouts comparable in the first place. Reality-gap diagnosis pays off only when it is tied to a closed-loop contract that names the simulator assumptions, the hardware measurements, and the alignment rule that says two rollouts are comparable. Without that contract, a model can look capable in a notebook while failing the first time a sensor drops a frame or a controller saturates.

The graduate-level habit is to separate three claims. The modeling claim explains which part of \(P_{\text{sim}}\) approximates \(P_{\text{real}}\). The systems claim explains which observation, action, or timing interface exposes the approximation error. The evidence claim records which paired rollout would convince a skeptical builder.

Which gap dominates which task family

Once those three claims are separated, the next practical question is where to spend the diagnostic budget, and the answer is that the six gap types are not equally likely across task families. Observation gaps dominate in vision-driven tasks. There, rendered textures, lighting, and depth noise differ from real camera output. Transition gaps dominate in tasks with deformable objects, liquids, or complex rigid-body chains, because simulator physics engines simplify these most aggressively. Actuator gaps dominate in high-speed or high-torque tasks, where the simulated motor model omits back-EMF (the voltage a spinning motor generates that opposes the driving current, reducing available torque at high speed), thermal derating, and current limits. Timing gaps dominate in any fast closed-loop task where the real controller adds latency that the simulated step function ignores. Contact gaps dominate in tasks requiring precise force regulation. Evaluation gaps dominate whenever the simulation success criterion captures a proxy state (object within 5 cm of goal) rather than the operationally correct state (object stable and graspable by the next agent in the pipeline).

What does it look like when the wrong gap type is targeted? A team invests two weeks adding visual realism to a simulator for a peg-insertion task, only to discover on the first hardware rollout that the culprit was actuator delay the whole time.

Evaluation mismatch causes silent failures on hardware. A policy can earn high simulator reward while producing unusable outcomes on the robot. Consider a gripper that barely touches an object at the goal position: the simulator awards full reward, but the object lands in a pose the downstream manipulator cannot grasp. A lax termination criterion invites reward hacking (the policy finding a shortcut that maximizes the reward signal without accomplishing the intended task). The simulator checks one scalar distance threshold at one timestep, so the policy learns the minimum action that crosses that threshold, not the sustained, stable configuration the real task demands. On hardware, the same action satisfies the proxy threshold and fails every downstream check. Identifying which family a task belongs to before training is the fastest way to choose the right transfer strategy.

A robust implementation starts with a paired rollout schema that logs inputs, outputs, units, timestamps, controller limits, termination reasons, and one failure label. Simulator and robot must emit the same artifact shape; otherwise the comparison is a story stitched from separate experiments.

1. Write a one-paragraph reality-gap contract with simulator assumptions and hardware measurements.
2. Choose paired trace fields that can be captured in both places without manual interpretation.
3. Run one deterministic smoke test and one perturbation test before scaling policy training.
4. Save a single artifact containing configuration, seed, metrics, videos or state logs, timing traces, and failure labels.
5. Compare repairs only when one script evaluates them on the same task panel and hardware protocol.

When a transfer attempt fails, avoid labeling the whole policy as weak. First assign the failure to observation, transition dynamics, contact, actuator delay, controller saturation, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.