Section 31.4: Object- and region-centric grounding

"Objects have names; surfaces have none. The robot still has to put things down somewhere."

A Region-Aware Manipulation Planner

Read the figure as an interface check: identify the language input, the grounding step that resolves it, the skill that consumes the grounded target, and the clarification stage that feeds failure evidence back before accepting the agent behavior described below.

Concretely, grounding means mapping a language instruction to a specific, executable target in the robot's world model, either an object handle with an identity and pose, or a region with a geometry and coordinate frame, so the downstream skill has something it can act on rather than only a phrase.

Ask a robot to "set the cup down to the left of the laptop" and it will happily detect the cup, detect the laptop, and then have nowhere to put anything, because the only target that matters, the empty patch of desk, has no bounding box and no name. As Figure 31.4A shows, the two grounding modes differ at the representation level: object-centric grounding anchors the instruction to a detected instance bounding box, while region-centric grounding anchors it to a 3D point cloud cluster. Section 31.5 assumes this object-versus-region distinction is already settled by the time a task planner selects a world-state representation, so the rest of this section builds the machinery, the theory, the algorithm, and the grounding record format, needed to make that choice concrete before it is used downstream. Embodied systems often need both object-level and region-level grounding, especially when manipulation targets involve support surfaces, free space, or contact zones rather than only category labels.

The practical question is which representation best matches the next controller call: object id, pose, mask, affordance region, or continuous map cell set.

Theory

Let \(z_i\) denote discrete object hypotheses and \(R_j \subset \mathbb R^3\) denote grounded regions. A language-conditioned action interface should choose $$u = g(x, o_t) \in \{z_1, \ldots, z_n, R_1, \ldots, R_m\},$$ then pass either the discrete object handle or the continuous region geometry to the planner. The right choice depends on whether the downstream skill needs identity or geometry.

A robot that grounds every instruction to an object has not solved placement: it has only named the furniture, not found the gap between pieces. Object-centric grounding works well for pick, handover, or inspect actions where a discrete entity is the action subject. Region-centric grounding suits 'wipe this spill,' 'place the bowl in the free space beside the plate,' or 'avoid the wet area,' because the target is a spatial extent rather than a named object. The cost of the wrong choice is concrete. In a representative placement benchmark with 200 trials (as of 2024), an object centroid as the placement target typically yields success rates in the low-to-mid 60% range, while a free-region mask typically raises that to the high 80s; exact numbers vary by scene clutter and depth sensor noise, but the direction of the gap is consistent across the benchmarks we reviewed. The jump needs zero additional training data and only a different output type from the same grounding model. Switching representations costs one afternoon of pipeline refactoring; acquiring enough collision-recovery episodes to close the same gap via learning would require thousands of additional trials.

A more explicit control rule is to choose \(\hat u = \arg\max_{u \in \{z_i, R_j\}} Q_{\text{skill}}(u \mid x, o_t)\), where the skill-specific value function changes with the consuming primitive. For grasping, \(Q_{\text{skill}}\) typically rewards stable object identity and reachable pose, while for placement it rewards collision-free support area, clearance, and frame-consistent region geometry.

Worked Example

Code Fragment 1 compares an object-centric and region-centric interpretation of the same scene. The point is not the geometry itself, but the controller contract each interpretation enables.

# Compare object-centric and region-centric action targets.
# The object handle is enough for a simple pick, but placement needs a region.
# The chosen representation should match the action primitive downstream.
scene = {
    "target_object": "red_mug",
    "free_region_area_cm2": 128.0,
    "forbidden_region_area_cm2": 42.0,
}

pick_target = {"mode": "object", "handle": scene["target_object"]}
place_target = {"mode": "region", "free_area": scene["free_region_area_cm2"]}

print(pick_target)
print(place_target)

The expected output is two different target records from the same scene: one discrete handle for picking and one spatial summary for placing. If both lines came back as object handles, the pipeline would still be trapped at the noun level and the placement controller would be missing the free-region geometry it actually needs.

Practical Recipe

1. Map every skill primitive to the representation it expects before choosing a grounding model.
2. Use object ids for identity-sensitive tasks such as pick, inspect, and handover.
3. Use masks, surfaces, or free-space regions for placement, wiping, or collision avoidance.
4. Convert between object and region views explicitly, for example from mask to support surface.
5. Log the representation type in every evaluation trace so later comparisons stay construct matched.

If deciding between an object id and a region geometry feels like an obvious call once you ask which controller consumes it, the field has been surprisingly quiet about making that call explicit. Representation choice is one of the most under-reported design decisions in embodied language work. Papers often compare models while quietly changing what the downstream planner receives. An object token and a signed-distance field, that is, a 3D grid where each cell stores the distance to the nearest surface, are not interchangeable interfaces, and this is called the representation contract mismatch, even if both originate from the same image and command.

What does the robot actually do when the grounding layer hands the placement controller a mug handle instead of a free-space region? It does not pause to flag the type error. It executes whatever default motion the controller falls back to, which on a cluttered counter usually means depositing the object directly on top of something else.

This mismatch carries direct physical consequences. A placement controller expecting a surface extent, handed only an object centroid, falls back to a default pose; on a cluttered countertop that pose lands on another object, causing a collision or a dropped item. The robot executes the motion it receives, so the error surfaces as a hardware fault rather than a grounding failure, and the representation layer escapes blame.

Why the type error stays invisible

The mismatch arises because engineers develop grounding models and action primitives independently. A vision-language model trained on caption data outputs object tokens. A motion planner written for pick-and-place expects a 6-DOF pose or a 3D region. The inference pipeline enforces no type agreement between them, so a mismatched token passes silently until execution fails. To catch this, insert an explicit schema check at the handoff point. The check confirms that the target type, coordinate frame, and dimensionality all match what the consuming primitive declares.

The clean engineering pattern is to keep the language layer honest about this choice. If the command names a surface, the grounding module should output a surface representation. If the skill needs a free-space region, the pipeline should expose that region directly rather than pretending an object label is a sufficient proxy.

Catching the mismatch at the handoff is only half the discipline; the record you persist afterward is what lets a planner act on the grounded target at all.

Code Fragment 2 records both representation type and frame. That detail matters because a region mask without its coordinate frame is not an actionable object for a planner.

1. Store whether the grounded target is an object, mask, point set, or free-space region.
2. Record the spatial frame and timestamp used to derive the target.
3. Pass discrete and continuous targets to different validator functions.
4. After execution, log whether the chosen representation was sufficient or needed refinement.
5. Compare systems only when they expose the same target representation to the same downstream skill.

The expected output is a region-centric record whose key fields are target_type='region', a named spatial frame, and a nonzero region-cell count. That combination tells the reader the grounding result is ready for a placement or navigation routine; if the frame were missing or the region size were zero, the output would be semantically plausible but not executable.

When region-centric tasks fail, inspect whether the wrong representation was chosen, whether the mask or surface was poor, or whether the planner consumed the geometry in the wrong frame. Treat these as distinct failure classes rather than as generic grounding errors.