Section 13.2: Visual, physics, sensor, and task randomization

"Vary the texture and the policy stops memorizing appearances. Vary the mass and it stops memorizing forces. Vary both and it has to learn physics."

A Domain Randomization Engineer

What This Section Builds

Two policies train on the same warehouse, the same robot, and the same task, yet one needs 50 real trials to transfer and the other needs 500: the only difference is that the first varied four separate axes of the world while the second varied just the pixels. This section makes that four-axis taxonomy operational, showing how to keep image variation, physical dynamics, sensor models, and task layouts separate in the manifest while still sampling coherent episodes.

The goal is to avoid two common errors: randomizing visual factors when the real problem is contact, and randomizing dynamics while leaving the sensor model cleaner than the robot's camera, depth stream, or joint encoder. Task randomization gets the deepest worked treatment of the four axes later in this section, since it is the axis builders most often skip, not because it matters less.

Theory

Figure 13.2A shows all four axes acting on a single pick-and-place scene at once, which is the situation every real deployment presents. Visual factors change the observation distribution: texture, lighting, background, object color, reflections, blur, and occlusion. Physics factors change transition dynamics: mass, friction, restitution (how much kinetic energy a collision preserves, where a low value means an object stops dead on impact and a high value means it bounces), damping, motor gain, delay, and contact solver tolerance.

Sensor factors corrupt measurement: intrinsics (a camera's internal parameters, such as focal length and lens distortion, that map 3D points to pixels), extrinsics (the camera's position and orientation relative to the robot), quantization (rounding a continuous measurement to the nearest discrete step a sensor can represent), dropout, rolling shutter (an artifact where a camera captures a frame row by row rather than all at once, distorting fast-moving objects), depth holes, latency, and encoder noise. Task factors change the semantic problem: object category, goal pose, distractors, clutter, start state, and success tolerance. The practical consequence is sharp. Empirical studies (as of 2024) typically report that a policy trained with all four axes randomized transfers in under 50 real trials, in practice, while the same policy that randomizes visual factors alone can require more than 500 trials to reach the same success rate. Every unexpected contact force or sensor dropout triggers a new failure mode that simulation never covered. Figure 13.2B lays out how the four axes feed a single closed-loop policy, with each axis targeting a distinct failure channel.

A policy trained on varied textures but fixed friction has learned to see, not to act. The real world corrects that assumption the moment contact begins. The important detail is coherent joint sampling: coupling. A shiny object should shift both appearance and grasp friction. A camera moved sideways should change intrinsics or extrinsics and the visible occlusion pattern. A heavier object should change acceleration, slip, and controller effort. Independent sampling is easy, but coherent sampling makes synthetic episodes physically teachable. In one tabletop grasping study, switching from independent to coupled mass-friction sampling cut the real-world trials needed to reach 80 percent success from roughly 4,000 to 600, because the policy stopped wasting capacity on physically impossible combinations that never appear on a real table.

Consider a specific case: a tabletop grasping episode samples object mass uniformly from 50 g to 500 g and, in the same draw, scales friction from 0.3 to 0.7 proportionally, because a denser object realistically has a larger contact patch. A policy trained on this coupled distribution learns to correlate grip force with perceived object size. An incoherent version that samples mass and friction independently can pair a 500 g object with friction 0.3, a combination that rarely occurs with plastic consumer goods on a clean table. The policy then hedges: it applies high grip force even for light objects, which causes rebound oscillation on the physical robot where object compliance differs from the simulator default.

Worked Example

For a tabletop push task, the following snippet records one coherent episode sample. Notice that each factor carries a class label, so the later failure analysis can separate camera noise from contact uncertainty and task layout.

# Sample one coherent randomization manifest for a tabletop episode.
# The class label keeps visual, physics, sensor, and task factors auditable.
from dataclasses import dataclass

@dataclass
class FactorSample:
    factor_class: str
    name: str
    value: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

episode = [
    FactorSample("visual", "albedo_shift", "matte blue object"),
    FactorSample("physics", "block_table_friction", "0.52"),
    FactorSample("sensor", "depth_dropout_rate", "3 percent"),
    FactorSample("task", "goal_offset_cm", "(4, -2)"),
]

for sample in episode:
    print(f"{sample.factor_class}: {sample.name} = {sample.value}")

Practical Recipe

1. Group candidate factors by class: visual, physics, sensor, and task.
2. Give each factor a unit, range, distribution, and plausibility source.
3. Write coupling rules for factors that should move together, such as camera pose and occlusion.
4. Reserve held-out factor combinations for evaluation, including rare but plausible combinations.
5. Report aggregate success and failure labels by factor class, not only by total episode count.

Of the four axes, the task axis is the one builders most often leave fixed, so it rewards a closer look before the full manifest comes together. Task randomization matters in embodied AI because a real robot faces a distribution of goals, object arrangements, and scene layouts that no fixed training scenario can represent. A policy trained on a single goal pose memorizes a reach trajectory rather than learning a goal-conditioned strategy; when the first deployment site places the target bin 8 cm to the left, the policy fails not from sensor error but from planning brittleness. Physical consequences compound: a robot that overshoots a mismatched goal pose can collide with a shelf, overload a joint, or drop a fragile object.

What happens when the robot is trained on a single fixed goal pose and the deployment bin is shifted 8 cm to the left? The answer is not a graceful adaptation: the policy overshoots, clips the shelf edge, and drops the object. The mechanism behind that failure is conditional policy coverage (whether the training distribution of goals was wide enough for the policy to have learned a rule that generalizes, rather than a single memorized trajectory).

Why task variation forces goal-conditioning

At each episode start, a sampler draws a goal pose, object set, clutter count, and start configuration. To succeed across that spread, the policy must condition its action on the current goal rather than replay a fixed trajectory. Wider task distributions force the network to encode goal information explicitly in its hidden state, and that representation is, in practice, the part most likely to transfer when the deployment site matches no single training configuration.

With the task axis now in view alongside the other three, one concrete platform shows all four at once. Each class earns its place by closing a different loop on real hardware. On a Franka Panda running a MuJoCo-trained push policy, visual factors stress the RGB perception head (the policy stalls when warehouse sodium lighting tints the table amber); physics factors stress prediction and control (a 0.52 versus 0.18 table-friction mismatch makes the block slide past the goal); sensor factors stress state estimation (RealSense D435 depth holes on glossy surfaces corrupt the grasp-point regression); and task factors stress planning (an 8 cm goal-bin shift exposes a memorized trajectory). The OpenAI Dactyl in-hand cube reorientation result is the canonical demonstration that randomizing all four together, not perception alone, is what carried a Shadow Hand policy from MuJoCo to physical hardware.

The graduate-level habit is to separate three claims. The coverage claim says the sampled factors overlap deployment. The coherence claim says sampled combinations could occur together on a real platform. The evidence claim records which factor class explains the improvement or remaining failure.

A robust implementation starts with a manifest schema that can survive scale. The same FactorSample record from Code Fragment 1 extends naturally into an evidence log so one script can compare a visual-only run, a physics-only run, a sensor-only run, and a combined run on the same task panel.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

Expected output: the printed trace should expose the tool, factor classes, held-out combination, metric, and failure breakdown. If one of those fields is missing, the example is not yet an evaluation artifact.

When a combined randomization experiment fails, do not treat the combined score as the diagnosis. Run ablations by factor class, then rerun the most suspicious coupled combination. This pattern distinguishes insufficient visual coverage from impossible physics, corrupted sensor modeling, and task-layout leakage.