Section 28.7: Scene representations for robotics: SLAM, real2sim, manipulation

"Perception earns its keep when the next action gets safer, faster, or easier to debug."

A Patient Embodied AI Agent

Problem First: Why This Representation Exists

A warehouse arm reaches confidently into the spot where a box used to be, fingers closing on empty air, because the only scene model it trusted was a photoreal render captured ninety seconds and one pick ago. A static computer-vision system can stop when it names an object or produces a clean visualization; an embodied system cannot. The robot needs a representation that is tied to coordinates, uncertainty, latency, and an action interface, because a late or uncalibrated result can be more dangerous than no result at all.

For this section, the useful mental model is the perception-to-action contract. The perception module receives sensor evidence, estimates a compact state, exposes confidence and timing, and lets a planner or controller decide whether the next action is allowed. This is the bridge from coordinate frames and sensor estimation to the closed-loop evaluation discipline used in sim-to-real transfer.

Figure 28.7.1 is the handoff diagram for scene representations in robotics: sensor evidence, geometric representation, uncertainty, latency, and action consumer each form a separate failure point. Figure 28.7.2 below then routes a single physical event through three concrete representations named there for the first time: an ESDF (Euclidean Signed Distance Field, a grid that stores at every point how far it is from the nearest obstacle so a planner can test free space cheaply) for collision checking, a scene graph for object relations, and a 3DGS (3D Gaussian Splatting, a scene representation that renders a photoreal view by blending many small colored, oriented blobs called Gaussians instead of a mesh or voxel grid) splat for visualization.

Mathematical Core

A robot scene memory is best viewed as a set of query functions, not a single universal format.

Different tasks ask different queries. A renderer asks for appearance, a planner asks for free space, a manipulator asks for contact geometry, and a language planner asks for object relations. A mature system routes each query to the representation that can answer it safely. (The worked miniature below shows this routing in a runnable form before the real2sim pipeline that depends on it is introduced, so read the code before the real2sim section that reuses its ESDF and scene-graph objects.)

Worked Miniature

Routing each query to its proper owner is not academic. Walk through a single shelf capture and watch the representations diverge under one physical change. The routing table divides semantic labor across representations: a visually rich model can own operator rendering while collision checking and grasp planning still demand geometry with stricter safety meaning.

Take a concrete case. A warehouse mobile manipulator captures a 3-second RGB-D (color plus per-pixel depth, the standard output of a depth camera) sweep of a shelf. The visual-inertial odometry front-end (NVIDIA Isaac ROS Visual SLAM) reports camera pose to within 8 mm at 30 Hz. From the same sweep, the arm planner updates an ESDF at 5 Hz and 3 cm voxels for a conservative free-space buffer, while a 3DGS model trained offline on 200 reference images drives the 60 fps operator feed. Shift a box 15 cm after a pick and the ESDF converges in 200 ms, one sensor cycle; the splat keeps painting the old pose until a re-capture fires. The arm planner stays safe because it queries the ESDF, not the splat. Read pose from the splat instead and the planner inherits a 15 cm error until the next full scan, enough to collide on the next pick attempt.

# Scene-memory routing table: dispatch each robotics query to the right representation
import numpy as np

# Simulated scene state: each representation owns a subset of queries
class ESDFMap:
    """Conservative signed-distance field for collision checking."""
    def __init__(self, voxel_res=0.03):
        self.voxel_res = voxel_res
        # Toy 3-D grid: positive = free, negative = occupied, zero = surface
        self.grid = np.ones((20, 20, 10), dtype=np.float32) * 0.5
        self.grid[8:12, 8:12, :] = -0.05  # box obstacle

    def query_free(self, xyz):
        ix = int(xyz[0] / self.voxel_res) % 20
        iy = int(xyz[1] / self.voxel_res) % 20
        iz = int(xyz[2] / self.voxel_res) % 10
        dist = float(self.grid[ix, iy, iz])
        return {"sdf": dist, "is_free": dist > 0.0, "rep": "ESDF"}

class SplatRenderer:
    """3D Gaussian splat stub: owns only the visual render query."""
    def query_render(self, camera_pose):
        # In a real system this would invoke the 3DGS rasterizer
        return {"rep": "3DGS", "fps": 60, "note": "visualization only"}

class ObjectGraph:
    """Scene graph: owns object state and relations."""
    def __init__(self):
        self.objects = {
            "box_A": {"pose": np.array([0.24, 0.24, 0.0]), "class": "box",
                      "grasped": False, "fresh": True},
        }

    def query_object(self, name):
        obj = self.objects.get(name, {})
        return {"rep": "SceneGraph", **obj}

def route_query(query_type, esdf, splat, graph, **kwargs):
    """Dispatch each robotics query to its owner representation."""
    if query_type == "avoid_collision":
        return esdf.query_free(kwargs["xyz"])
    elif query_type == "render_operator_view":
        return splat.query_render(kwargs.get("camera_pose"))
    elif query_type == "plan_grasp":
        return graph.query_object(kwargs["object_name"])
    else:
        raise ValueError(f"Unknown query: {query_type}")

# Build the scene memory stack
esdf  = ESDFMap(voxel_res=0.03)
splat = SplatRenderer()
graph = ObjectGraph()

# Three different consumers ask three different questions
queries = [
    ("avoid_collision",      {"xyz": np.array([0.24, 0.24, 0.0])}),
    ("render_operator_view", {"camera_pose": np.eye(4)}),
    ("plan_grasp",           {"object_name": "box_A"}),
]

for q_type, q_args in queries:
    result = route_query(q_type, esdf, splat, graph, **q_args)
    print(f"{q_type:25s} -> rep={result['rep']:12s}  result={result}")

avoid_collision           -> rep=ESDF          result={'sdf': -0.05, 'is_free': False, 'rep': 'ESDF'}
render_operator_view      -> rep=3DGS           result={'rep': '3DGS', 'fps': 60, 'note': 'visualization only'}
plan_grasp                -> rep=SceneGraph     result={'rep': 'SceneGraph', 'pose': array([0.24, 0.24, 0. ]), 'class': 'box', 'grasped': False, 'fresh': True}

Note what plan_grasp actually returns: the object's pose, class, and a grasped flag, not a full mesh. A grasp planner consumes that record by combining it with the ESDF's local geometry around the object's pose to select a contact point and approach direction, then flips grasped to True and marks the record stale for any other consumer once the gripper closes. This is the concrete mechanism by which "manipulation" in this section's title is implemented: contact state lives in the scene graph, contact geometry is borrowed from the ESDF at query time, and the two are never merged into one file.

Routing queries to the right live representation keeps a deployed robot safe. The same discipline decides whether a robot can be trained at all, which is where real2sim transfer enters the pipeline. Real2sim transfer matters because a hand-crafted simulator forces you to specify every object geometry, mass, and friction by hand. On a real shelf with dozens of novel products, that is not feasible. Without real2sim, adding one novel product to the training set typically costs on the order of several hours of a roboticist's time to model manually in CAD, depending on shape complexity and the fidelity required; with a real2sim pipeline, the same object enters simulation from a 3-second RGB-D sweep. Without real2sim, the policy trains on geometry that does not match the physical object, and sim-to-real transfer fails at contact: the simulated grasp succeeds while the physical grasp slips or misses entirely. A policy trained on geometry that does not match the physical world is not a manipulation policy: it is a fiction that performs well only in the simulator that invented it.

The transfer works in three steps. First, export the SLAM-registered point cloud or mesh to a collision-ready format: OBJ (a plain-text mesh format most simulators can import directly) or USD (Universal Scene Description, Pixar's format for composing geometry, materials, and physics properties into one editable scene). Second, attach physical properties as metadata: mass from a scale reading, friction from a slip test. Third, a simulator importer (MuJoCo's obj2mjcf or Isaac Sim's USD pipeline) instantiates the scene and replaces default parameters with the measured values. The scene now carries geometry and dynamics anchored to a specific real-world capture.

Debugging And Evaluation

Evaluate each representation inside the same action loop that will consume it, logging the sensor stream, calibration version, frame transform, and failure label so the comparison is construct-matched. For the full debugging protocol, see section 28.6. The key addition for multi-representation systems is verifying that the routing table itself is logged: when a query is dispatched to the wrong owner (for example, a grasp planner reading from a splat rather than the ESDF), the failure label must capture the mis-route, not just the downstream contact error.