Section 12.2: Manipulation: ManiSkill3, robosuite, RoboCasa, robomimic, RLBench

"A manipulation score tells you how well the agent grasped objects in these scenes, with this action space, under this success criterion. Change any one and you are measuring something else."

A Manipulation Benchmark Auditor

What This Section Builds

Swap a robosuite controller string from OSC_POSE to JOINT_VELOCITY and the very same policy on the very same lift task can lose twenty points of success rate, even though nothing about the robot's skill has changed: that single line of config is the difference between two papers' headline numbers. This section makes manipulation benchmarks operational, teaching how to read a tabletop result as a claim about contact, vision, demonstration learning, object generalization, task generalization, or scene diversity. As Figure 12.2A illustrates, a manipulation score is only credible when the object, task, demonstration, controller, and observation boundaries are made visible rather than left implicit.

The goal is not to memorize benchmark names. The goal is to design a comparison where ManiSkill3 speed, robosuite controller choices, RoboCasa scene diversity, robomimic datasets, and RLBench task variation are treated as evidence boundaries rather than interchangeable labels.

Theory

A manipulation benchmark samples an initial object arrangement, exposes robot observations, accepts actions through a controller, and scores whether task predicates (the pass/fail rule the simulator checks, such as "object height above table exceeds 0.04 m") become true. The same task name can change meaning when the action space changes from end-effector deltas to joint torques, when RGB observations become privileged state (ground-truth object pose and velocity handed directly to the policy instead of inferred from pixels), or when demonstrations come from a different controller; the controller mechanics behind that last point are unpacked in the next two paragraphs. Figure 12.2B traces this pipeline stage by stage and shows which suite specializes which stage.

Controller choice matters because physical robots have joint limits, torque saturation (the motor can no longer supply the commanded force), and contact dynamics. These properties amplify or suppress policy errors differently depending on the control interface. An operational-space (end-effector delta) controller hides joint-level complexity behind an inverse-kinematics layer. This makes grasping easier to learn, but it masks whether the policy can handle singularities (arm configurations where the inverse-kinematics solution becomes unstable or undefined) or payload variation. A joint-torque controller exposes the full mechanical chain, so policies must reason about inertia and cable tension. Those challenges determine real-robot reliability. A benchmark result under one interface does not transfer to the other, and real deployments nearly always differ from the interface used during evaluation.

An operational-space controller computes a desired Cartesian end-effector pose. It calls the robot's inverse kinematics solver to get target joint angles, then passes those targets to a low-level PD controller (a proportional-derivative feedback loop that converts a position error into a corrective torque) running at 500 Hz or higher. A joint-space controller skips that layer entirely. It commands torques or positions directly on each joint, so the policy must respect the robot's kinematic chain explicitly. robosuite exposes both modes through a unified wrapper. Swapping the controller string changes the action dimensionality, the action scale, and the physical meaning of every output dimension. So you must freeze the controller name in the result artifact.

Matching the Split to the Claim

The split must match the claim. For imitation learning, separate training and test demonstrations so trajectory memorization is not counted as skill. For object generalization, hold out object instances or categories. For task generalization, hold out task templates, not merely random seeds for the same template.

Worked Example

Code Fragment 1 shows a compact leakage audit for manipulation. If the training and test panels share object identifiers, task templates, or demonstration identifiers, the reported success rate may reflect familiarity rather than manipulation competence.

# Audit manipulation splits for object, task, and demonstration leakage.
# The result should be empty before a manipulation benchmark supports
# a generalization claim about held-out episodes.
from dataclasses import dataclass

@dataclass(frozen=True)
class ManipulationEpisode:
    task: str
    object_id: str
    demo_id: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

train = {
    ManipulationEpisode("lift", "mug_001", "demo_017"),
    ManipulationEpisode("stack", "cube_002", "demo_021"),
}
test = {
    ManipulationEpisode("lift", "mug_009", "demo_110"),
    ManipulationEpisode("stack", "cube_002", "demo_222"),
}

train_objects = {episode.object_id for episode in train}
test_objects = {episode.object_id for episode in test}
print(f"object_leakage={sorted(train_objects & test_objects)}")

Practical Recipe

1. Choose the suite that matches the claim: demonstrations for robomimic, task variety for RLBench, controller studies for robosuite, GPU-parallel training for ManiSkill3, or household-style manipulation for RoboCasa.
2. Freeze train, validation, and test splits by task template, object instance, scene, and demonstration identifier.
3. Use the same observation keys, action space, controller, horizon, and success predicate for every method in the comparison.
4. Report per-task and per-seed success, not only a mean that hides brittle tasks.
5. Label failures as perception miss, grasp/contact failure, controller saturation, task-predicate miss, recovery failure, or split leakage.

Consider what this looks like in practice. A team trains a diffusion policy on robomimic's human demonstrations for the lift task and reports 94% success. A second team trains a transformer policy in ManiSkill3 on the same task name and reports 81% success. The numbers appear comparable, but the first uses privileged object-pose observations and 200 human demonstrations recorded at 20 Hz with a specific gripper, while the second uses RGB-D (Red-Green-Blue-Depth) images and 50,000 environment rollouts. The 13-point gap says nothing about which policy architecture is better; it measures the protocol gap, not the algorithm gap. In practice, published leaderboards for manipulation often mix these conditions, and readers who do not check the protocol detail column risk treating the numbers as construct-matched evidence when they are not.

Once that experiment boundary is sharp enough to name every field in the self-check, the payoff becomes clear: manipulation benchmarks become useful when they make hidden choices visible. A policy trained on robomimic demonstrations may be excellent at reproducing dataset actions but weak under new object placements. A policy trained in ManiSkill3 may benefit from high-throughput exploration but still need an object and task split that tests transfer. A robosuite controller comparison may be valid only inside the chosen action interface.

The graduate-level habit is to ask what each suite contributes to the evidence. ManiSkill3 and Isaac-style GPU workflows scale rollout counts, robosuite makes controller choice explicit, RoboCasa and BEHAVIOR assets broaden household variation, robomimic fixes offline demonstration evaluation, and RLBench tests task-level variation. A paper should name which of those dimensions it measured.

Why Each Suite Was Built

Each suite exists because a specific research question was underserved. ManiSkill3 was designed to answer questions about sample efficiency at scale: when you need millions of rollouts to learn contact-rich behavior, CPU-serial simulators become the bottleneck, so GPU-parallel environments eliminate that bottleneck and let the research question be about the policy, not the infrastructure. The scale difference is concrete: a contact-rich task that requires 50,000 environment episodes to reach 80% success on a CPU-serial simulator can reach the same threshold in roughly 300 episodes when the GPU-parallel renderer fills a batch with 4,096 environments simultaneously, because each gradient step sees a far wider slice of the contact distribution. robosuite was designed because controller choice had been treated as an implementation detail rather than a scientific variable; making it explicit and modular lets a single paper compare joint-space and operational-space control on identical tasks.

Before reading on: if a policy achieves 90% success on a fixed tabletop with three familiar objects, what do you expect its success rate to be in a kitchen with varied layouts, distractor items, and appliances it has never seen? The answer in practice is much lower, and that gap is exactly the motivation for the suites below. RoboCasa addresses the narrowness of tabletop-only benchmarks: real kitchens have layout variation, distractor objects, and diverse appliances, so a policy that works on a fixed table with three canonical objects has not been tested on the actual distribution. robomimic exists because most imitation-learning papers before it trained and evaluated on the same demonstrations without a held-out set, making reported numbers unrepeatable; its standardized dataset releases and evaluation protocol convert a reproduction problem into a comparison problem. RLBench targets task-level generalization: its 100-plus task variants, each with multiple language phrasings and visual configurations, are calibrated so that a policy cannot succeed by recognizing a single fixture arrangement. Knowing why each suite was built tells you immediately when to use it and when its design assumptions do not match your research question.

A robust manipulation benchmark starts with a split file. The split file should list task templates, object IDs, scene IDs, demonstration IDs, and seeds. The evaluation runner should consume that file for every method so a new policy cannot quietly choose easier episodes.

1. Choose one primary suite and write the task contract in that suite's native terms.
2. Freeze task, object, scene, demonstration, and seed splits before hyperparameter tuning.
3. Evaluate baselines and candidates with the same observation keys, action space, controller, and horizon.
4. Save per-episode success, reward, horizon length, contact or grasp status, and video trace.
5. Aggregate by task family and seed so one easy task cannot dominate the conclusion.

Code Fragment 2 turns the manipulation protocol into a result artifact. The important field is split_name, because it keeps a reported success rate attached to the held-out condition it actually measured.

# Record a manipulation result with the split and controller attached.
# A success rate without these fields is not comparable across suites
# or even across two runs from the same suite.
from dataclasses import dataclass, asdict

@dataclass
class ManipulationResult:
    suite: str
    split_name: str
    controller: str
    observation: str
    seeds: tuple[int, ...]
    success_rate: float

    def as_row(self) -> dict[str, object]:
        return asdict(self)

result = ManipulationResult(
    suite="RLBench",
    split_name="heldout_task_variations",
    controller="end_effector_delta_pose",
    observation="front_rgb+proprioception",
    seeds=(10, 11, 12, 13, 14),
    success_rate=0.62,
)
print(result.as_row())

A frozen result artifact tells you what was measured, but it does not tell you why a low number came out the way it did, and that is where failure localization takes over. When a manipulation experiment fails, localize the failure before changing the model. Replay the episode and tag whether the failure came from perception, grasp approach, contact instability, controller saturation, recovery, task-predicate scoring, or a held-out split mismatch. This prevents a model change from masking an evaluation problem.

A policy that scores perfectly in simulation but collapses on hardware is not a strong policy; it is a well-rehearsed one that never met the real stage.