Section 32.3: Vision-language encoders and open-vocabulary detection

"Naming an object outside your training set is impressive. Telling the arm where to grasp it is the part that matters."

A Robot That Needed Boxes, Not Labels

Say "green bottle" to a classical detector trained on 80 fixed classes and it shrugs; say it to an open-vocabulary detector and it draws a box around the right object even though "bottle" was never a labeled category it was trained on. That is the leap this section is about: locating objects named by free-form text rather than by a closed list of training labels. The proposal-then-verify pipeline turns such a text query into a verified, world-frame action target, and a grounded box stays a hypothesis until geometry confirms it. Read the figure as an open-vocabulary detection contract. The model may name objects outside a fixed label set, but the robot still needs boxes, masks, frame transforms, confidence calibration, and a rejection path for ambiguous detections. This section has two halves: it first recaps how the vision-language encoder from section 32.2 turns a region and a text query into comparable embeddings, then it spends the remainder building the detection, scoring, masking, and verification machinery that turns that embedding comparison into a box, a mask, and finally a world-frame target a robot can act on.

What Open-Vocabulary Detection Actually Produces

A grounded detector does not return "the answer." It returns a set of candidate regions conditioned on language. A prompt such as "red mug" yields boxes or points that are semantically relevant. It yields them often before the system knows whether the object is graspable, visible enough for tracking, or safe to approach. The scale difference is stark: a COCO-trained (COCO, or Common Objects in Context, is the standard benchmark dataset that fixes detectors to 80 predefined object categories) closed-vocabulary detector recognizes 80 fixed categories, while a CLIP-based (CLIP, or Contrastive Language-Image Pre-training, is the encoder introduced in section 32.2 that aligns images and text in a shared embedding space) open-vocabulary detector can ground any of the roughly 400,000 nouns in a standard English lexicon without retraining. This jump reshapes the training budget. A team at a logistics warehouse needed roughly 50,000 labeled training episodes to teach a closed-vocabulary detector each new seasonal SKU class. After the team switched to an open-vocabulary backbone, the same coverage required about 300 prompted examples per category.

That distinction matters because detectors sit upstream of planning. They form what practitioners call the language-conditioned proposal stage, much like how A* and sampling planners propose routes that still have to be checked against dynamics and safety constraints.

Scoring Boxes And Masks

If detection only ever yields candidates that must be verified, the first practical question is how those candidates get ranked in the first place, which is exactly what the scoring rule decides.

A standard grounding stack scores each candidate box \(b_i\) for text query \(q\) with a detection score and a language-alignment score. A simple decision rule is

followed by thresholding, non-maximum suppression, and optional mask refinement. In practice the multiplicative form tends to be more robust than either term alone, because it penalizes boxes that are visually dubious even when the language alignment is high. After a box is selected, a promptable segmentation model can turn it into a mask \(m_i\) (a per-pixel label of which pixels belong to the object, as opposed to the box's coarse rectangle), which is often what the robot actually needs for contact planning or free-space reasoning.

Viewed formally, this is a constrained optimization step under uncertainty: maximize the grounded score over proposals while enforcing overlap, visibility, and reachability constraints before the policy is allowed to act. The detector is therefore part of a state-space estimation pipeline (a system that tracks a belief about the world and updates it as new sensor evidence arrives, rather than a one-shot answer), not merely a captioning front end.

You can also read the post-detection filter as a lightweight Bayes update (a rule for revising a probability estimate as new evidence arrives) with an implicit covariance budget (a working notion of how much positional uncertainty remains tolerable before the target is considered localized). Image-space evidence proposes hypotheses, geometric checks shrink the feasible set, and the final policy consumes only the survivors. Those survivors are the proposals whose uncertainty has dropped enough for action.

The final proposal set is therefore a filtered set \(\mathcal B^\star = \operatorname{NMS}(\{b_i : \text{score}(b_i \mid q) \ge \tau\})\), where NMS denotes Non-Maximum Suppression, the step that removes lower-scoring duplicate boxes whose overlap with a higher-scoring box exceeds a threshold, and that respects overlap suppression before any action module sees it. That algorithmic detail matters because embodied failures often come from duplicate or conflicting boxes surviving proposal selection, not from the language query alone.

For two proposals \(b_i\) and \(b_j\), the usual overlap test is \(\operatorname{IoU}(b_i, b_j) = \frac{\lvert b_i \cap b_j \rvert}{\lvert b_i \cup b_j \rvert}\), and NMS suppresses the lower-scoring box when \(\operatorname{IoU}(b_i, b_j) > \eta\). In robotics that threshold is not a cosmetic hyperparameter, because a too-low \(\eta\) can erase small manipulable targets while a too-high \(\eta\) can leave duplicate boxes that confuse the grasp selector.

Once a box \(b^\star \in \mathcal B^\star\) survives, the embodied handoff is still incomplete until the system derives a world-frame action target, for example \(\hat p = \Pi^{-1}(m^\star, D_t, T_{\text{camera}\rightarrow\text{world}})\) from the refined mask \(m^\star\), current depth map \(D_t\), and camera extrinsics. This makes the contract explicit: open-vocabulary detection proposes image-space regions, while action consumes geometry in a robot frame.

A minimal accept rule is therefore: execute only if \(b^\star = \arg\max_{b \in \mathcal B^\star} \text{score}(b \mid q)\), the mask support exceeds a visibility threshold \(\lvert m^\star \rvert \ge \kappa\), and the projected target satisfies clearance and reachability checks \(c(\hat p) \le 0\). Stated this way, the detector is one stage in a constrained pipeline rather than an oracle.

Written compactly, the pipeline is $$ B_0 = \operatorname{Detect}(I_t, q), \quad s_i = \text{score}(b_i \mid q), \quad B^\star = \operatorname{NMS}(\{b_i \in B_0 : s_i \ge \tau\}), \quad m^\star = \operatorname{Seg}(I_t, b^\star), \quad \hat p = \Pi^{-1}(m^\star, D_t, T_{\text{camera}\rightarrow\text{world}}). $$ That sequence makes the module boundary explicit: language chooses proposals, segmentation sharpens support, and geometry converts the final mask into something a controller can actually use.

Worked Example

Code Fragment 1 compresses that ranking stage into a toy calculation, which makes the selection rule concrete before a maintained GroundingDINO (an open-vocabulary detector that proposes boxes from a text prompt) or Grounded-SAM (GroundingDINO composed with the Segment Anything Model, or SAM, a promptable segmentation network) pipeline hides it.

# Rank grounded boxes by combining detector confidence and text alignment.
# This is the minimal decision rule before non-maximum suppression and masking.
# The robot should only pass top-ranked boxes to geometric verification.
import numpy as np

boxes = np.array(["left_box", "center_box", "right_box"])
det_scores = np.array([0.93, 0.74, 0.81], dtype=float)
text_scores = np.array([0.42, 0.89, 0.51], dtype=float)
joint_scores = det_scores * text_scores

best = int(np.argmax(joint_scores))
print({"selected_box": boxes[best], "joint_scores": joint_scores.round(3).tolist()})

The expected output is one selected region, center_box, together with three joint scores that show language evidence reshuffling the detector ranking. Reading the trace should make the logic legible: left_box was visually stronger on its own, but after text conditioning the center proposal becomes the correct candidate to pass into NMS, masking, and geometric verification.

Once the box is selected, the next question is whether the robot needs a rectangle or a pixel-accurate support region. For navigation or rough target selection, a box may be enough. For grasp planning, collision checking, or object-centric memory, a mask is usually much more useful.

Code Fragment 2 shows the maintained pattern at the point where most builders actually work.

# Ground a phrase to boxes, then refine the boxes into masks.
# pip install groundingdino-py segment-anything
# The detector proposes regions; the segmenter sharpens them for action.
image = load_image("tabletop_scene.png")
boxes, phrases = grounding_dino_predict(image, text_prompt="red mug", box_threshold=0.35)
masks = sam_refine_masks(image=image, boxes=boxes)

print({"num_boxes": len(boxes), "num_masks": len(masks), "top_phrase": phrases[0]})

The expected output is a small proposal set where the number of masks matches the surviving number of boxes and the top phrase remains tied to the user query. In a healthy GroundingDINO plus SAM-style pipeline, this tells the builder the semantic proposal stage and the geometric refinement stage stayed synchronized; if the counts diverged or the phrase changed, the handoff between detection and segmentation would need inspection.

Promptable Segmentation And 3D Follow-Through

Tuning those thresholds settles which boxes survive, but a surviving box is still a flat rectangle, and the moment the robot has to touch the object that rectangle stops being enough.

A box is rarely the end of the story. If the robot must grasp, avoid, or remember the object, it often needs the mask to intersect with depth. The mask can be lifted into a point cloud, used to fit a 3D extent, or stored as a memory key for later re-identification. This is one of the clearest bridges from Chapter 32 to occupancy and neural scene representations.

Why a mask beats a box for contact

Masks matter physically because a bounding box includes background pixels and occluding edges that corrupt depth estimates. A grasp planner using a box centroid can land on empty space beside a thin object; a pixel-accurate mask constrains the sampled point cloud to the object surface, giving the contact planner a stable estimate of extent and orientation.

A promptable segmenter such as SAM encodes the image once with a heavyweight encoder, then runs a lightweight mask decoder on the box prompt. The decoder returns three masks at increasing granularity (whole object, part, subpart), each scored. The robot picks the granularity its action needs: whole-object for grasping, part for insertion.

A grounded box is a hypothesis, not a destination; the robot earns the right to act only after geometry has audited the proposal and signed off. The most useful diagnostic artifact for this section is a four-panel record: original image, top grounded boxes with scores, selected mask over depth, and the final action decision. That artifact lets you see whether the failure arose from phrase grounding, box ranking, mask quality, or geometric follow-through. It also stops teams from reporting open-vocabulary success on screenshots while the physical robot still grasps the wrong object.