Section 42.4: Perception for manipulation

"A grasp starts as a perceptual claim."

A Systems Calibration Log

Ask a manipulation controller what it wants from vision and it will not ask for a class label. It wants to know which face of the object it can safely push against. The answer lives in a handful of geometric outputs: 6D pose (an object's full position and orientation in space, three translation and three rotation parameters), graspable surfaces, free-space channels, occlusion estimates, contact normals, and uncertainty fields.

This section walks through how each of those outputs is actually produced and used: the Theory section below derives the uncertainty-aware scoring rule that turns a pose distribution into a grasp ranking, the Algorithm box sequences the steps from raw sensor stream to a ranked, uncertainty-checked action, the Worked Example and Walkthrough show that ranking changing a real decision, and the Library Shortcut names the specific tools (FoundationPose, Contact-GraspNet, SAM 2) that produce these outputs on real hardware.

Those outputs bridge vision models and control. The main lesson is that manipulation perception should be judged by action utility under uncertainty, not by static detection scores alone. A perception system that scores well on benchmarks but leaves the controller blind to pose uncertainty is not a perception system for manipulation; it is a perception system for leaderboards.

Theory

Perception for manipulation is a structured estimation problem. The latent state includes object identity, object pose, free space, support relation, graspable contact patches, and confidence in each estimate.

The useful error metric is therefore downstream: how much does state uncertainty change grasp ranking, collision risk, or recovery timing? Manipulation perception is only good if the wrong estimate would actually change what the robot does. This is the core of action-conditioned perception evaluation, and it separates manipulation vision from generic object recognition.

The equation below makes this concrete: the first term marginalizes grasp success over the posterior pose distribution given the image \(I\) and depth \(D\), and the second term ranks grasps \(g\) by mean quality \(Q\) penalized by its variance under pose uncertainty, with \(\beta\) setting how strongly variance is punished. The \(\arg\max_g\) notation just means: evaluate this penalized score for every candidate grasp \(g\) and keep the one with the highest value, \(\hat g\).

$$ p(g \mid I, D) \propto \int p(g \mid x_o)\,p(x_o \mid I, D)\,dx_o,\qquad \hat g = \arg\max_g \mathbb{E}_{x_o}[Q(g, x_o)] - \beta\,\mathrm{Var}_{x_o}[Q(g, x_o)] $$

Worked Example

# Penalize grasps whose score is too sensitive to pose uncertainty.
grasps = [
    {"id": "g1", "mean_q": 0.86, "var_q": 0.07},
    {"id": "g2", "mean_q": 0.81, "var_q": 0.01},
    {"id": "g3", "mean_q": 0.75, "var_q": 0.03},
]

beta = 1.5
scored = []
for g in grasps:
    robust = round(g["mean_q"] - beta * g["var_q"], 3)
    scored.append((g["id"], robust))

scored.sort(key=lambda row: row[1], reverse=True)
print(scored)

Expected output: The expected ranking promotes the lower-variance candidate. That is the right behavior when manipulation failure is expensive and ambiguity can be reduced later by active sensing.

Practical Recipe

1. Calibrate multi-camera and robot frames before measuring pose quality.
2. Store grasp or affordance scores together with uncertainty, not as naked logits.
3. Use the same object ids across segmentation, pose estimation, and planner logs.
4. Add active perception motions when the top action depends strongly on occluded geometry.
5. Evaluate perception with action-conditioned metrics such as reachable grasp success or collision-free lift rate.

Walkthrough: Mug Pick from a Cluttered Shelf

A robot picks a mug from a cluttered shelf. The wrist-mounted depth camera produces a 640×480 point cloud at 30 Hz. A segmentation model (SAM 2) isolates the mug's visible surface, covering roughly 60% of its body because a book occludes the handle. FoundationPose then fits a 6D pose hypothesis set to that partial cloud; the top hypothesis has a translation uncertainty of ±8 mm along the x-axis (perpendicular to the shelf face). Eight millimeters sounds small. But a typical parallel-jaw fingertip contact patch is only 6 mm wide, so that single uncertainty interval is wider than the entire gripping surface. Treated as a point estimate it almost guarantees a miss. Instead, the grasp ranker turns it into a safe above-approach in under 150 ms.

The grasp planner queries two candidate approach angles: one from the front (score 0.83, variance 0.09, driven by handle occlusion) and one from above (score 0.77, variance 0.02). With a robustness penalty of β = 1.5, the above approach scores 0.77 − 0.03 = 0.74 versus the front approach scoring 0.83 − 0.135 = 0.695. The system selects the above approach, moves the wrist camera 15 cm upward, re-estimates pose with the handle now visible, and then re-ranks with reduced variance before executing the grasp. The full sequence is: segment, estimate with uncertainty, propagate uncertainty into scoring, decide whether to act or re-perceive, then act.

That decision to re-perceive rather than commit is exactly the behavior a benchmark of static image scores would never reward. Image metrics care about masks and classes; manipulation metrics care about whether the same estimate yields a stable approach, grasp, and recovery. The gap is measurable: in representative 2024 bin-picking evaluations, pose estimators above 90% detection AP (average precision, the standard accuracy score for detection benchmarks) produced 60–65% grasp success when uncertainty was ignored, and routing the identical estimates through an uncertainty-aware ranker lifted that to the low-to-mid 80s with no retraining.

From Estimate to Active Sensing

Perception for manipulation is also a natural entry point for active sensing. If the robot can move the wrist or camera to shrink uncertainty on the top-ranked grasp, the perception system has already become a planner partner rather than a frozen upstream block, a pattern covered more fully under active and embodied perception.

If the chosen action was bad, ask whether the state estimate was wrong, the uncertainty was ignored, or the planner consumed the estimate incorrectly. Manipulation perception failures often live at those interfaces.