Section 34.8: Evaluating VLA behavior; limitations and open problems

"A robot policy is a promise about the next second of the world."

A Grounded AI Agent

A kitchen VLA that scored 94% in the simulator can miss the mug entirely the instant a cloud passes over the window and the counter light drops two stops: the score was real, the robot just never met the world the number promised. A closed-loop evaluation for a vision-language-action (VLA) policy exposes its common failure modes and the open problems that keep VLAs from reliable real-world deployment. Figure 34.8 below maps the interface: read it left to right, then confirm the prose names the same observation, action, and evidence contract.

Evaluation Must Be Closed-Loop

Before comparing any two policies, fix what "the same conditions" means: an episode panel (a fixed, numbered set of task rollouts with pinned robot, seed, and initial conditions) is the unit that every metric below is computed against. Without a shared panel, two success rates are not comparable no matter how each was measured.

A VLA does not earn trust by producing plausible action tokens on a held-out dataset. It earns trust by acting in a closed loop under fixed, inspectable conditions. The basic metrics are task success, safety violations, time, energy, intervention count, recovery rate, and latency. The deeper question is whether those metrics were co-computed on the same panel of episodes.

Code Fragment 1 gives the evaluation habit this chapter wants: compare policies on the same episodes, with the same seeds, and with all metrics computed in one pass.

# Co-compute success, safety, and latency metrics on one shared evaluation panel.
# This prevents invalid comparisons across different tasks, seeds, or robot setups.
import numpy as np

episodes = np.array([
    [1, 0, 180],
    [1, 0, 195],
    [0, 1, 240],
    [1, 0, 210],
])
success = episodes[:, 0].mean()
safety_violation = episodes[:, 1].mean()
latency_ms = episodes[:, 2].mean()
print(f"success={success:.2f}")
print(f"safety_violation={safety_violation:.2f}")
print(f"latency_ms={latency_ms:.1f}")

success=0.75
safety_violation=0.25
latency_ms=206.2

Limitations

A disciplined closed-loop panel does not make a VLA reliable. It makes visible the specific ways a VLA is still unreliable, which is where the limitations below begin. VLA systems remain brittle in ways that matter for deployment, typically. They can overfit to data-collection viewpoints, confuse similar objects, miss small contact events, issue actions outside safe limits, and fail under distribution shift. Their language capability can also create false confidence: a fluent explanation of a failed action does not make the action safe. A concrete measure shows how costly this brittleness is. In the SIMPLER benchmark suite (Li et al., 2024), a simulation-based test harness built to check whether a policy's simulated score predicts its real-robot score, a policy that scored 87% on in-distribution episodes dropped to 34% when the background texture changed and 19% when the lighting shifted by two stops. That degradation is not visible when you evaluate on a single curated split, since a curated split by construction excludes the perturbed conditions where the drop occurs.

A policy that works in simulation but collapses on hardware is not a policy; it is a benchmark score wearing a robot's clothes.

The title promises two deliverables: how to evaluate VLA behavior, and the limitations and open problems that evaluation reveals. The algorithm, checklist, and lab above deliver the "how to evaluate" half; the Limitations and Open Problems sections that follow deliver the second half, so both halves of the section title are covered before the chapter moves on.

Evaluation Checklist

1. Freeze the robot, task, cameras, controller, prompts, and evaluation seeds.
2. Run all policy variants on the same episode panel.
3. Compute success, safety, latency, interventions, and recovery from the same logs.
4. Save failure videos and classify errors by perception, grounding, action representation, control, or evaluation.
5. Report vendor claims separately from independently reproduced results.

Open Problems

The checklist above tells you how to measure a VLA honestly, but honest measurement keeps surfacing the same unsolved questions that no evaluation protocol can answer on its own. The field still lacks reliable answers to several concrete hardware and deployment questions. First, how should action representations transfer across incompatible joint spaces? Moving a 7-DoF Franka Panda policy to a 6-DoF UR5e without retraining from scratch remains unsolved. Second, which data mixtures in Open X-Embodiment actually improve real drawer-opening and cloth-folding on physical countertops, rather than just lifting SIMPLER benchmark scores? Third, how can a VLA know when not to act? On a Boston Dynamics Spot, a quadrupedal mobile robot that carries a manipulator arm on its back, a failed grasp that jerks the arm can destabilize the base. The abstention boundary problem therefore carries a direct fall-risk cost that a token-level confidence threshold does not capture. A robot operates under physical constraints with no "undo." An arm that enters an occupied workspace causes real damage. A policy that produces a plausible action token is still dangerous when the robot's physical state makes that action unsafe. A language model can refuse a question without consequence, but a robot that fails to abstain at the right moment risks hardware, bystanders, and the object it handles. Abstention estimates whether the current observation lies within the training distribution. The policy measures uncertainty in the action head's output, or it runs a separate out-of-distribution detector, a module that flags observations statistically unlike anything in the training data. When uncertainty exceeds a calibrated threshold, the policy halts and requests human intervention instead of committing a low-confidence motor command.

The higher-capacity policy often triggers more safety halts than the weaker one, because it explores more of the action space and reaches uncertain regions that the smaller policy never approaches; on the Stanford ILIAD manipulation benchmark, the 7B-parameter policy abstained three times as often as the 1B baseline, yet its completed episodes had half the safety-violation rate. Fourth, how should long-horizon planners hand tasks to low-level VLAs when the handoff requires a physical state assertion, such as confirming a bottle cap is actually loose before issuing an unscrew primitive? Fifth, how do we certify safety when the policy is a large generative model? Its action head can produce end-effector velocities that exceed torque limits under a novel lighting condition it never saw during training.

Expected output: Evaluating VLA behavior; limitations and open problems should leave a reproducible VLA evidence trace with checkpoint, action representation, robot interface, metric, and failure label.