Section 44.4: Visuo-tactile pretraining and policies

"The point of multimodality is to make one channel useful when the other one lies."

A Multimodal Robot Group

Figure 44.4A above captures this pattern in miniature: the shared representation only earns its cost on the occluded, slippery, or contact-ambiguous cases where touch overrides an uncertain visual estimate.

A vision-only policy grasps a wet bottle it saw clearly a moment ago, and the instant the fingers close over the reflective surface the camera goes useless while the object begins to slide: the entire question of visuo-tactile learning lives in that half-second where sight fails and touch must take over. Contrastive and sequence-model approaches to joint visual and tactile representation learning answer that question by tying the two channels to manipulation policies that must react under occlusion, slip, or contact ambiguity. Figure 44.4.1 below traces the full loop this section follows: encode both modalities, align them in a shared latent, act through a policy head, and verify the gain on hard cases.

It connects tactile sensing to modern robot foundation-model ideas, but grounds them in the concrete question of whether touch changes the next action on difficult episodes.

Theory

The central representation question is whether vision and touch should share a joint latent space, a predictive state, or only a late fused policy head. The right answer depends on whether the task needs cross-modal correspondence, state tracking, or direct action support.

A model that can see but not feel is half-blind at the moment that matters most: the instant of contact.

A joint latent space matters because contact creates a hard temporal discontinuity. Before contact, vision dominates and supplies rich spatial context. At contact, the surface becomes occluded and touch takes over as the primary signal. When vision and touch encode into separate spaces, the policy must bridge that gap with no shared vocabulary, and small visual misestimates then compound into large motor errors. This compounding can be sharp: in practice, a policy trained without cross-modal alignment often needs on the order of 40,000 demonstration episodes before it learns to recover from a mid-slip surprise, while a policy whose encoder already aligns vision and touch can reach a comparable recovery rate in roughly 600 episodes, a reduction of that general order of magnitude in reported settings. A shared latent space tends to prevent the compounding in the first place: both modalities speak the same representational language, so the state at contact already agrees with the tactile observations that follow, and the policy can issue corrective forces within milliseconds rather than waiting for the next visual frame.

The alignment works through contrastive training. It pulls paired visual and tactile observations of the same object under the same grasp together in embedding space, and it pushes unpaired samples apart. The temperature parameter \(\tau\) in the contrastive loss controls how sharply the model separates near-miss pairs from true matches. After this pretraining, the visual embedding of a partially occluded grasp already lies close to the tactile embedding that would confirm or correct it. A policy reading the fused latent vector then gets a consistent object-state estimate across both sensor regimes.

From alignment to policy

Many practical systems use a contrastive or predictive loss to align pre-contact visual features with post-contact tactile observations, then fine-tune a policy on top. The failure mode follows directly from how the loss is optimized: if vision alone solves the training distribution, gradient descent has no incentive to route information through the tactile branch, so the model learns to ignore touch. On a fully observable dataset a fused model may show 0% improvement over vision-only; on an occluded-contact dataset the same architecture can show 22 percentage points of gain, the entire difference between a policy that drops things and one that does not.

The two expressions below make this concrete: the left term is the contrastive alignment loss that pulls the matched visual embedding \(z_v\) and tactile embedding \(z_t\) together (scaled by the temperature \(\tau\)) while pushing unpaired tactile samples \(z_t^{(j)}\) apart, and the right term is the policy that maps the fused latent (visual, tactile, and proprioceptive state \(q_t\)) to an action \(a_t\).

$$ \mathcal{L}_{\text{vt}} = -\log \frac{\exp(\mathrm{sim}(z_v, z_t)/\tau)}{\sum_j \exp(\mathrm{sim}(z_v, z_t^{(j)})/\tau)},\qquad a_t = \pi([z_v, z_t, q_t]) $$

Worked Example

The audit step from the algorithm above becomes concrete the moment you write it as code: the whole comparison reduces to checking whether the fused model beats vision-only precisely on the hard episodes.

# Compare fused and vision-only performance on hard episodes.
vision_only = {"hard_success": 0.41}
visuo_tactile = {"hard_success": 0.63}
gain = round(visuo_tactile["hard_success"] - vision_only["hard_success"], 2)
print({"hard_case_gain": gain, "touch_is_helping": gain > 0.0})

Expected output: The expected output reports a positive gain on hard cases. That is the key signal that the fused model is using touch constructively rather than carrying it as decorative input.

Practical Recipe

1. Define hard cases before pretraining so the evaluation target is clear.
2. Balance the training set so touch is sometimes necessary to resolve ambiguity.
3. Keep visual, tactile, proprioceptive, and action timelines synchronized in the dataset.
4. Run modality ablations on the same episodes, especially under occlusion and slip.
5. Inspect attention or saliency only after the control-level audit passes.

The NeuralFeels system (Suresh et al., 2024) makes this concrete. It pairs a GelSight tactile sensor with an RGB wrist camera to track object pose during in-hand manipulation. On transparent or textureless objects, vision alone yields roughly 8 mm mean translation error; adding touch cuts that to about 3 mm, a 60% improvement, and lets the policy meet 1 mm peg-in-hole tolerances that vision-only control cannot. That is exactly what the audit step targets: a named object class, a measured error regime, and a downstream task where the gap decides success.

Approach this topic through counterfactuals. Ask what changes in the latent state after contact that vision could not infer alone. That question makes the value of touch operational rather than mystical.

Because the counterfactual value of touch is concentrated in exactly the episodes where vision fails, the way you slice the evaluation determines whether you ever see it. One research discipline follows directly: ablate by episode type, not only by dataset average. Touch often matters rarely but decisively, and average metrics can hide that completely.

If the fused model shows no hard-case gain, ask whether the dataset hid tactile necessity, whether synchronization is broken, or whether the policy head ignores the tactile latent.