Section 31.3: Grounding language in perception; referring expressions

""The red cup" is not a description. It is a challenge: find exactly one object that satisfies it in the scene in front of you, right now."

A Grounding Module Under Pressure

Read the figure as an interface check: trace the four stages, language input, grounding evidence, executed skill, and clarification fallback, before accepting the agent behavior described below.

Say "the red mug on the left" to a robot facing three red mugs, two of them on the left, and the single hardest engineering problem in the room is not the words but the commitment: the machine must collapse that phrase to exactly one segmentation mask, act on it, and be wrong in a way you can audit if it picked the second-best mug. This section builds the resolver that turns such a phrase into one grounded object, scores attribute and relational evidence separately, and signals uncertainty instead of guessing when the evidence is underdetermined.

A referring expression is a phrase that picks out one specific object in the current scene (for example "the red cup" or "the mug beside the kettle"). It is not a label; it is a live query against the world the robot can see right now, and a robot that cannot answer it cannot act.

Grounding language in perception is exactly this: turning the words of a referring expression into a probability distribution over the objects a perception system currently reports, then committing to the single object with the highest score. The rest of this section builds that scoring pipeline step by step, from the perceptual candidate set through attribute and relational evidence to the clarification fallback when the evidence does not decide the case.

Embodied agents resolve names, attributes, and relations such as color, position, containment, and ownership into specific entities visible to the robot. On a Franka Panda arm running an Open X-Embodiment policy (a pick-and-place control policy trained on the large cross-robot Open X-Embodiment dataset), "the red mug on the left" must collapse to one segmentation mask in the wrist-camera frame before MoveIt (the standard ROS motion-planning library that turns a target region into a collision-free arm trajectory) can plan a grasp. On a Boston Dynamics Spot navigating a warehouse, "the pallet near the loading door" must resolve against a live RTAB-Map (a real-time SLAM library that builds and maintains a persistent map of mapped entities) of entities the robot may have last seen ten seconds ago from a different pose.

The practical question is how to combine visual evidence with relational language when several objects share the same category or attribute, for example three identical red mugs on a cluttered table where only the relation to a kettle distinguishes the target, and the kettle itself may be occluded by the gripper at the moment of grasp.

Theory

Given objects \(z_1, \ldots, z_n\) extracted from perception and a referring expression \(x\), the grounding problem is $$\hat z = \arg\max_i \; p(z_i \mid x, o_t) \propto p(x \mid z_i, o_t)\, p(z_i \mid o_t).$$ The perceptual prior says what objects are present; the language likelihood says which of those objects best matches the expression.

Relational language makes the problem harder because the referent depends on other objects. In 'the mug beside the kettle,' the target score depends on two things: the mug's own features and the probability that a nearby kettle exists and is correctly localized. This is the cascaded anchor dependency problem. One missed detection in the scene graph (the set of detected objects and their spatial relations, the perceptual structure a relational referring expression is checked against) collapses the relational term to zero and silently demotes the correct referent (the algorithm below implements the anchor-confidence check, step 4, that catches this case before it reaches the robot's arm). In practice, the scale of the benefit is large, though the exact numbers depend on scene clutter and detector quality: on a cluttered table with four similar mugs, a purely unary model (one that scores each candidate only on its own attributes, such as color or category, with no relation to other objects) typically needs on the order of thousands of labelled examples to learn which mug humans usually mean, while adding one well-detected relational anchor can reduce that requirement by an order of magnitude, because the relation eliminates most of the competing candidates before training even begins. Grounding therefore inherits every weakness of the detector and every ambiguity of the language model.

Worked Example

Code Fragment 1 scores three candidate objects against color, category, and spatial relation. It is small, yet it exposes the same reasoning pattern used by larger grounding models and dialogue systems.

# Resolve a referring expression using attribute and relation evidence.
# Each candidate receives unary scores and a relation score to the kettle.
# The selected object is the one with the highest combined grounding score.
candidates = [
    {"id": "obj_1", "label": "mug", "color": "red", "near_kettle": True},
    {"id": "obj_2", "label": "mug", "color": "blue", "near_kettle": True},
    {"id": "obj_3", "label": "bowl", "color": "red", "near_kettle": False},
]

scores = {}
for obj in candidates:
    unary = 0.7 if obj["label"] == "mug" else 0.1
    color = 0.4 if obj["color"] == "red" else 0.0
    relation = 0.5 if obj["near_kettle"] else -0.2
    scores[obj["id"]] = round(unary + color + relation, 2)

print(scores)
print(max(scores, key=scores.get))

The expected output is a score table where obj_1 beats obj_2 by a modest relation-driven margin, followed by the winning referent id. That pattern tells the reader the grounding model used the kettle relation rather than only the noun mug; if obj_2 won instead, the likely failure is missing relational evidence rather than missing category evidence. The numbers make the stakes concrete: without the relational term, obj_1 and obj_2 are tied at 1.1 each and the robot has no basis to choose; with it, the gap opens to 1.6 vs 1.2, a 33% margin that is enough to act on. One missing anchor collapses that margin back to zero.

Practical Recipe

1. Detect or segment a candidate set before asking the language model to choose among them.
2. Score attributes and relations separately so you can diagnose which signal failed.
3. Keep the candidate set visible to the planner instead of passing only the winning object id.
4. When the top two candidates are close, trigger a clarification question or an active view change.
5. Re-ground after any action that changes visibility, object pose, or scene layout.

That ability to name the eliminating evidence and reason about its disappearance is exactly what marks grounding as a perception problem rather than a language one. Referring expressions are a clean example of embodied partial observability. The agent may know the words but not the full scene, or it may see the scene but lack one relational anchor. Good systems therefore maintain uncertainty over the referent rather than forcing a premature point estimate.

This is also why closed-loop evaluation matters. A model with strong top-1 accuracy is still a poor embodied component if it never signals uncertainty, because it never triggers clarification or camera motion. Grounding is scored by the final action outcome, not the static matching score.

Code Fragment 2 records the grounding result as an auditable object. The important fields are the chosen referent and the score margin over the runner-up, because that margin should drive clarification or active sensing.

1. Store the candidate list, the winning id, and the runner-up gap in one record.
2. Tie the winner to the current camera frame or map timestamp so stale groundings are detectable.
3. Route low-margin groundings to clarification or view-planning instead of execution.
4. Log whether the downstream action preserved the intended relation after grasp or motion.
5. Benchmark on scenes with distractors, occlusion, and viewpoint change, not only on clean still images.

# Build the auditable grounding record and decide clarify vs. execute.
# CLARIFY_THRESHOLD is the confidence margin tau from the algorithm above.
CLARIFY_THRESHOLD = 0.5

record = {
    "expression": "the red mug beside the kettle",
    "frame_timestamp": 1729.442,
    "candidates": {"obj_1": 1.48, "obj_2": 1.12, "obj_3": 0.32},
    "winner": "obj_1",
    "runner_up": "obj_2",
}
record["margin"] = round(record["candidates"][record["winner"]]
                          - record["candidates"][record["runner_up"]], 2)
record["decision"] = "clarify" if record["margin"] < CLARIFY_THRESHOLD else "execute"

print(record)

The expected output is a compact grounding artifact with a winner, a runner-up, and a margin small enough to trigger clarify rather than execute. In a live system this exact trace is what separates a calibrated referential agent from an overconfident one: the same top prediction is preserved, but the action gate changes because the uncertainty is still operationally significant.

From Gate To Diagnosis

Once that margin is logged as the gate between clarify and execute, the same record becomes the starting point for diagnosis when a grounding does go wrong. When grounding fails, separate detector failures, relation failures, stale-memory failures, and confidence-calibration failures. Different fixes apply to each: better proposals, explicit relation modeling, view planning, or threshold tuning.