Section 11.4: NVIDIA Isaac Sim and Isaac Lab

"Sometimes the hard part is not stepping physics. It is making the world look, sense, and scale like the task."

A Sensor-Rich Robot Learner

From Isaac Gym To Isaac Lab

Isaac Gym was important because it showed how GPU simulation could change reinforcement learning throughput. The current path is Isaac Lab. NVIDIA's migration documentation states that Isaac Gym Preview Release, IsaacGymEnvs, OmniIsaacGymEnvs, and Orbit have been superseded by Isaac Lab workflows. For new projects, the practical rule is direct: start with Isaac Lab unless you are reproducing an older result.

What Isaac Lab Adds

Isaac Lab is not only a physics engine wrapper. It organizes environments, assets, sensors, controllers, task definitions, vectorized simulation, and learning libraries into a reproducible workflow, and as Figure 11.4A illustrates, the stack is strongest when assets, sensors, randomization, training traces, and replayable evaluation stay connected. This matters for the chapters that follow: benchmarks in Chapter 12, domain randomization in Chapter 13, and massively parallel RL in Chapter 17.

The main engineering benefit is that task structure becomes explicit. A good Isaac Lab experiment separates scene assets, robot articulation (the kinematic chain of rigid links and joints that make up the robot's body), observation groups, action terms, reward terms, termination rules, curriculum events, and randomization events. That separation is why the stack is heavier than MuJoCo, and also why it is useful when one policy must be trained across thousands of sensor-rich variations. Figure 11.4B below makes that layering explicit, showing how Isaac Lab sits on top of Isaac Sim and OpenUSD, with legacy Isaac Gym positioned as the deprecated predecessor beneath both.

Of the three foundations in that vocabulary, OpenUSD is the one whose role is least obvious, so it is worth seeing exactly why a scene format earns a place beside the simulator and the learning framework. OpenUSD matters for embodied AI because physical constraints depend on what the scene actually contains. Friction coefficients, mass distributions, collision geometry, and articulation joints must transfer intact across three stages: the authoring tool, the simulator, and the real robot's planning stack. When each tool uses its own format, those properties drop out or map wrongly. A policy then trains in one environment and faces a different physical world during deployment. USD provides a shared contract. The same asset carries its physics annotations into Isaac Sim, a renderer, a motion planner, and a data pipeline, and no one re-specifies them at each boundary.

USD stores scene content as a layered composition of typed primitives. Each object, mesh, joint, or material is a "prim" with attributes and metadata. Layers stack non-destructively: a base layer holds geometry, a physics layer adds mass and friction, and a randomization layer overrides those values per training variant. When Isaac Lab loads a task, it resolves that layer stack into a final scene graph. PhysX (NVIDIA's GPU-accelerated rigid-body physics engine, the dynamics solver underneath Isaac Sim) then receives the composed physics attributes directly. Any application that reads USD resolves the same stack, so the geometry the animator exports is the geometry the simulator collides against.

What happens when a friction coefficient silently drops to zero at the format boundary between your asset tool and your simulator? The robot trains for days in a frictionless phantom world, then slips on the first real surface it touches. USD's layered composition exists precisely to close that gap.

A Task Config Mental Model

Once the scene and its layered physics are composed, the next question is how Isaac Lab organizes everything the policy must observe and act on. Isaac Lab examples usually revolve around task configuration: scene, robot, observations, actions, rewards, terminations, events, and randomization. Code Fragment 1 uses a lightweight dataclass to show this structure without requiring a local Isaac install.

# Isaac Lab task planning skeleton: separate task choices before training.
# Dataclasses make the environment contract explicit and easy to audit.
# Replace this with Isaac Lab config classes in a full project.
from dataclasses import dataclass, asdict

@dataclass(frozen=True)
class TaskContract:
    robot: str
    observations: tuple[str, ...]
    actions: tuple[str, ...]
    randomizations: tuple[str, ...]
    success_metric: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

reach_task = TaskContract(
    robot="Franka Panda",
    observations=("joint_pos", "joint_vel", "target_pose"),
    actions=("joint_position_targets",),
    randomizations=("object_mass", "table_friction", "camera_pose"),
    success_metric="end_effector_distance < 0.03 m",
)

print(reach_task)

TaskContract(robot='Franka Panda', observations=('joint_pos', 'joint_vel', 'target_pose'), actions=('joint_position_targets',), randomizations=('object_mass', 'table_friction', 'camera_pose'), success_metric='end_effector_distance < 0.03 m')

When Isaac Lab Is The Right Default

Choose Isaac Lab when the task needs thousands of GPU environments, rich sensors, USD assets, synthetic data, or a path toward sim-to-real on NVIDIA hardware. It is strongest for locomotion, manipulation, and humanoids, where rendering and sensor fidelity matter as much as dynamics.

Consider a specific case: the Unitree H1 humanoid locomotion task in the Isaac Lab example suite runs 4096 parallel environments on a single A100, stepping physics at 200 Hz per environment. At that scale, a 10-minute training run accumulates roughly 500 million physics steps, an amount that would take weeks on a single-threaded CPU simulator. The same task in MuJoCo on CPU (single environment, 200 Hz) would take on the order of thousands of hours to collect equivalent data (rough illustrative estimate; actual throughput depends on hardware and task complexity).

That gap is why the tool choice is not about personal preference. A task may require curriculum learning over varied terrain heights, masses, and friction coefficients at once. There, throughput determines feasibility, not preference. Consider a concrete case. A curriculum needs 50,000 episodes to converge on a single-threaded CPU simulator. On 4,096 parallel GPU environments it converges in roughly 300 episodes of wall-clock time, because each "episode" the researcher waits for really runs 4,096 episodes at once.

Do not treat the stack as one opaque score. Validate dynamics, observations, and training throughput separately. A policy that learns quickly from privileged state may still fail when camera latency, depth noise, lighting, or segmentation labels enter the observation loop. A policy that succeeds in simulation but collapses on real hardware is not a policy: it is a very expensive hypothesis about what the real world might have been like.

Choose a lighter tool when you need a small control experiment, quick model inspection, or a minimal reproducible dynamics loop. Heavy tools can hide mistakes if the team cannot inspect the physics contract.

Expected output: An Isaac Lab experiment record should include the task config, asset and USD references, observation groups, action space, reward terms, termination rules, randomization events, sensor settings, number of parallel environments, GPU device, and replayable evaluation seeds.