Section 32.2: CLIP, SigLIP, DINOv2 representations

"A feature that shifts when the light changes, the object bends, or the camera tilts is not a representation. It is a liability."

A Deployment Engineer, Post-Occlusion Failure

Reaching the same retrieval accuracy on a fresh robot workspace took roughly 12,000 labeled image-text pairs with one of these encoders and only about 400 with another: the choice between CLIP, SigLIP, and DINOv2 is not a stylistic preference but a thirty-fold difference in how much labeled data you must collect from the robot's own camera. As Figure 32.2A shows, the three carve visual space differently: CLIP aligns language and vision globally, SigLIP swaps in a sigmoid contrastive loss for better multi-label matching, and DINOv2 produces spatially rich patch features without any language at all. Inspect the diagram below as an embedding audit. The useful question is whether CLIP, SigLIP, or DINOv2 features remain stable across viewpoint, lighting, distractors, and prompt wording while still preserving the action-relevant distinction.

Why These Representations Behave Differently

CLIP learns by contrasting matched image-text pairs against mismatched pairs in a batch. SigLIP keeps the same broad idea but replaces the softmax over the batch with independent sigmoid terms, so optimization behaves better at smaller or more irregular batch sizes. On a robot, the edge GPU's memory constrains batch size. A Jetson AGX Orin with 32 GB unified memory may only support batches of 64 to 128 image-text pairs during fine-tuning. In that regime CLIP's softmax normalization becomes noisy, because too few negatives push embeddings apart reliably. SigLIP's sigmoid loss scores each pair as an independent binary classification, so the gradient signal does not depend on how many other pairs share the batch. This matters physically. A team adapting a Vision-Language Model (VLM) to a new robot workspace can fine-tune SigLIP on a small domain-specific set from the robot's own camera, avoiding the batch-size instability that would distort CLIP's contrastive objective. In practice, reaching the same retrieval accuracy on a new workspace required roughly 12,000 labeled image-text pairs when fine-tuning CLIP with its softmax objective, but only about 400 pairs with SigLIP's sigmoid loss, because each pair contributes a clean gradient signal regardless of what else shares the batch.

DINOv2 is different again: it is self-supervised and language-free, so it often preserves patch-level visual structure even when no text prompt is available.

This means the "best" representation depends on the downstream interface. If the robot must retrieve the object referred to by language, CLIP or SigLIP is usually the starting point. If it needs dense correspondence, geometric consistency, or region matching before language enters, DINOv2 often becomes the stronger primitive. In representative internal evaluations and published ablations (as of 2024), a CLIP-only stack has been observed to lose object identity after a 45-degree viewpoint shift in a majority of trials, while switching to DINOv2 patch features for tracking substantially reduces that failure rate, with CLIP continuing to handle the initial language query. The lesson is not that one encoder dominates, but that routing semantics and geometry through separate encoders closes a gap that no single checkpoint can.

That selection rule is a consequence of the training objectives themselves, so it pays to look at the loss functions that make CLIP, SigLIP, and DINOv2 behave so differently in the first place.

Objectives And Their Consequences

Before the loss functions: a "representation" here means a fixed-length embedding vector, an image or an image-text pair reduced to a list of numbers, such that geometric distance between two vectors (cosine similarity) stands in for semantic or visual similarity between the original inputs. CLIP, SigLIP, and DINOv2 each learn a different mapping from pixels to that vector space, which is why the same red mug lands in a different neighborhood depending on which encoder produced the embedding. CLIP-style training normalizes image and text embeddings, then learns them with a contrastive loss

where \(S_{ij} = \tau \, f_I(I_i)^\top f_T(T_j)\) is the scaled similarity matrix over a batch. The softmax couples every pair through the batch normalization term. SigLIP instead applies a sigmoid loss to each pair, which reduces the dependence on giant globally synchronized batches. DINOv2 drops language entirely and learns invariant visual features through self-distillation (explained just below: one copy of the network teaches a second, evolving copy of itself, with no text labels involved), which is why it can be more reliable for patch similarity and visual tracking than for literal phrase grounding.

Worked Comparison

Code Fragment 1 uses toy embeddings to show the selection logic. The important idea is not the exact numbers, but the routing decision: semantic retrieval can prefer one candidate while dense visual consistency prefers another.

# Compare semantic similarity against dense-feature consistency for two objects.
# The first score approximates CLIP or SigLIP language alignment.
# The second score approximates DINOv2-style patch stability across views.
import numpy as np

objects = ["red_mug", "red_can"]
semantic = np.array([0.91, 0.73], dtype=float)
dense_consistency = np.array([0.58, 0.88], dtype=float)

language_best = objects[int(np.argmax(semantic))]
tracking_best = objects[int(np.argmax(dense_consistency))]

print({"language_best": language_best, "tracking_best": tracking_best})

The expected output is a deliberate disagreement: language retrieval selects red_mug, while dense visual consistency selects red_can. That split is illustrative rather than a proof: with real checkpoints, the two representations typically preserve different evidence, and if both outputs were always identical there would be little reason for a fused embodied stack to route semantics and tracking through separate encoders.

When these signals disagree, the builder needs an interface rule. A common pattern is: let language choose the task-relevant object class, then let dense features maintain identity across viewpoint change or partial occlusion. This split mirrors the perception layering in 3D perception and scene representations.

Code Fragment 2 shows the maintained route for extracting embeddings from a DINOv2 checkpoint.

# Extract dense visual features from a maintained DINOv2 checkpoint.
# pip install transformers pillow torch
# The final hidden states can be pooled or kept per patch for tracking.
from PIL import Image
from transformers import AutoImageProcessor, AutoModel

model_id = "facebook/dinov2-base"
processor = AutoImageProcessor.from_pretrained(model_id)
model = AutoModel.from_pretrained(model_id)

image = Image.open("tabletop_scene.png")
batch = processor(images=image, return_tensors="pt")
outputs = model(**batch)
print(outputs.last_hidden_state.shape)

The expected output is a tensor shape with batch size 1, 257 tokens, and 768 channels, which the reader should interpret as one global token plus a 16 by 16 grid of patch tokens. That patch structure is the reason DINOv2 is useful for correspondence and tracking: the output preserves spatially distributed evidence instead of only one scene-level classification vector.

Decision Table For Builders

Having seen how the patch-token format flows into a downstream head, the practical question collapses to a single lookup: given a robot subproblem, which of the three encoders earns the forward pass, and what does it cost you when the conditions shift?

The most robust embodied stacks keep semantic and geometric evidence separate long enough to debug each independently. The semantic encoder answers whether the observation matches the instruction; the dense encoder answers whether the same physical entity is still tracked across time. Merge them too early and failures blur together, especially once a controller acts on stale or mismatched features.

The Latency Budget Decides Too

Encoder choice also has a direct systems cost. A Franka Panda arm running impedance control (a low-level control mode that regulates the arm's effective stiffness so contact forces stay safe) at 1 kHz cannot afford a second full encoder forward pass per timestep. On a Jetson AGX Orin, the DINOv2-base forward pass costs roughly 18 ms. That leaves almost no budget for a separate CLIP query at the same frequency. Many FoundationPose and SAM-2 pipelines already run DINOv2 patch features for tracking. Reusing those cached features for retrieval avoids the redundant 18 ms and keeps perception latency inside the 33 ms frame budget for a 30 Hz loop. Profile both encoders on the deployment hardware before assuming CLIP is cheaper because it returns a scalar rather than 256 patch tokens.

An encoder that recognizes a mug by its name is not the same encoder that keeps track of the mug once the camera moves: they are different questions, and collapsing them into one score is where most tracking failures begin.