"A VLA policy is a contract between language grounding, image tokens, robot state, and action decoding."
A Grounded AI Agent
This section assumes familiarity with vision-language model architecture from section 32.1 and with action chunking from section 22.2. If you already understand how a transformer encodes image and text tokens jointly, you can skim the first subsection and start at "The Interface Contract." The VLA policy formulation introduced here is extended in section 34.2 (training objectives and data) and section 34.3 (inference-time control loops), and the action-head design recurs in Part 5 alongside diffusion and flow-matching policies.
A VLA can name an object and still fail the motion. Always evaluate grounding, action accuracy, latency, and recovery as separate properties.
A robot arm hovers over a cluttered table. You say "hand me the blue mug." The model identifies the mug, localises the handle, plans a grasp, and moves the arm, all from a single forward pass through a transformer trained on internet video and robot demonstrations. That is not science fiction: it is what production VLAs do as of 2024-2025, and it is why the field shifted so sharply after 2022. Vision-language models already cracked open-ended understanding; adding an action head turns that understanding into physical behaviour. Figure 34.1A sketches that architecture end to end: vision and language encoders feed a cross-modal transformer whose output the action head projects into a continuous end-effector trajectory. In this section you will see exactly where that boundary sits, what the action head commits to, and why every semantic claim the model makes must survive the much harsher test of real-world control.
Why VLMs Are Not Enough
The core idea of this section in one sentence: a VLA is a VLM plus an action head, so the entire "VLM to VLA" story is the story of what that one added component must guarantee, timing, physical limits, and a committed motor trajectory, that a captioning model never had to guarantee.
Ask a state-of-the-art vision-language model to describe a cup on a table and it will do so flawlessly; ask it to actually pick the cup up and it will emit words while the arm never moves. That gap is the whole subject of this section: embodiment adds a requirement no captioner faces, namely that the answer must change the world. If the instruction is "put the cup on the coaster," the system must decide where the cup is, where the coaster is, which grasp is feasible, how the arm should move, when the gripper should close, and how to recover if the cup slips. This is the gap between knowing and doing, and bridging it requires an action head. A model that can describe a grasp without executing one is a spectator, not an agent.
A Vision-Language-Action (VLA) model extends the multimodal stack by adding an action channel. Formally, it learns a policy \(\pi_\theta(a_{t:t+H} \mid o_{1:t}, q, r_t)\), where \(o\) is visual observation, \(q\) is the language instruction, \(r_t\) is robot state, and \(a_{t:t+H}\) is an action chunk over a short horizon. The action representation varies across systems: discrete tokens in RT-1 and RT-2, diffusion or flow outputs in RDT and pi-zero, and compressed action tokens in FAST-style autoregressive policies (which predict actions one token at a time, left to right, like a language model predicting words).
Why Predict a Chunk, Not a Step
Predicting a chunk of \(H\) steps rather than a single step matters because robot servos run at 100 to 500 Hz while a VLA inference pass takes 100 to 300 ms. Without a buffered chunk, the arm stalls between policy calls. A chunk also lets the policy commit to a smooth trajectory shape, reducing jerk at grasp contact. Mechanically, the action head outputs \(H\) delta poses in one forward pass. The low-level controller queues them, executes each at the servo rate, and discards the remainder when the next chunk arrives. The gap this closes is stark. A pure VLM prompted to move a robot arm achieves near-zero success on physical pick-and-place tasks (RT-2 ablations, Brohan et al. 2023). RT-2 is one of the concrete VLA systems this section returns to repeatedly; it is described in full below (see "Take a concrete case"), but the short version needed here is that it uses the same PaLI-X backbone (Google's large-scale vision-language transformer, pretrained on web image-text pairs) with only an action head added. It reaches 62% success on novel objects the model never saw in robot data, drawing on web knowledge already in the pretrained weights. Matching that generalization from scratch, using only task-specific robot demonstrations, would require roughly 130,000 episodes. RT-2 achieves it with the same 130 robot-data episodes it was fine-tuned on, because the pretrained backbone already carries the object semantics.
A common assumption is that replacing a VLM's text output with motor commands is a straightforward swap that inherits all of the VLM's language generalization automatically. This is wrong in the embodied AI context because action generation is subject to hard physical constraints that text generation never faces: the output must arrive within a fixed servo cycle, reference a robot state that may already be stale, and remain within workspace and joint-angle limits. The correct mental model is that the action head introduces an entirely new output contract, one that couples the model to timing, synchronization, and proprioceptive consistency, so generalization from web-scale language pretraining must be validated against these physical constraints separately, not assumed.
Language tells the policy what counts as progress, vision tells it what the world currently affords, and the action head commits to a motor trajectory. The action head is where a VLA stops being a scene interpreter and becomes an embodied policy.
The algorithm below traces the full path from raw inputs to executed motor commands, making explicit where the two action-head routes (discrete tokens versus continuous denoising) diverge and where physical limits are enforced.
Algorithm: VLM-to-VLA Grounding and Action Selection
Input: language instruction \(q\), visual observation sequence \(o_{1:t}\) (RGB frames), robot state \(r_t\) (joint angles, end-effector pose), policy parameters \(\theta\)
Output: action chunk \(a_{t:t+H}\) (motor commands over horizon \(H\))
- Encode each image frame \(o_i\) through the vision encoder to obtain patch token embeddings \(z_i^v \in \mathbb{R}^{N_v \times d}\).
- Tokenize the language instruction \(q\) through the text encoder to obtain token embeddings \(z^l \in \mathbb{R}^{N_l \times d}\).
- Concatenate all tokens into a joint sequence: \(z = [z^l; z_1^v; \ldots; z_t^v; z^r]\), where \(z^r\) is a linear projection of \(r_t\).
- Pass \(z\) through the cross-modal transformer to produce contextualized representations; retain the final hidden state \(h \in \mathbb{R}^d\).
- Determine the action head type: if discrete-token mode, project \(h\) onto per-dimension bin logits and decode by argmax, the operation that returns the index of the highest-scoring logit rather than a weighted blend of all of them, to get \(a_{t:t+H}\); if continuous mode, proceed to step 6.
- Sample initial noise \(\epsilon \sim \mathcal{N}(0, I)\) and run the denoising network \(D_\phi\), a learned model that starts from random noise and iteratively refines it toward a valid sample, the same mechanism used by image diffusion models, for \(K\) steps, conditioned on \(h\), to produce a smooth trajectory: \(a_{t:t+H} = D_\phi^{(K)}(\epsilon \mid h)\).
- Apply action normalization: \(\hat{a} = \mu_a + \sigma_a \cdot a_{t:t+H}\) using dataset statistics \((\mu_a, \sigma_a)\) to recover physical units.
- Check \(\hat{a}\) against workspace limits and joint-angle bounds; clamp any out-of-range dimension and record a violation flag.
- Send \(\hat{a}\) to the low-level controller, which executes commands at the servo rate (typically 100 to 500 Hz) while the policy runs at its own inference rate.
- Observe the next state \(r_{t+1}\) and frame \(o_{t+1}\); if the task success criterion \(\nabla_\theta \mathcal{L}\) has not been met and the episode horizon allows, increment \(t\) and return to step 1.
Checkpoint
So far: the algorithm turns image patches, instruction tokens, and robot state into one joint sequence, runs it through the cross-modal transformer, and then splits into two possible action-head routes (discrete-bin argmax or iterative denoising) before normalization and limit-checking produce the command actually sent to the controller.
Step-Through: discrete-token action decode
Trace one inference pass with tiny numbers. The instruction "pick up the green block" tokenizes into 4 text tokens; a 96x96 frame yields 36 patch tokens; robot state \(r_t\) adds 1 projected token, so the joint sequence has 41 tokens. The transformer produces a final hidden state \(h \in \mathbb{R}^{8}\). The action head projects \(h\) onto bin logits for one action dimension (the x-axis end-effector delta) using only 4 bins for clarity, with bin centres \([-0.04, -0.01, +0.01, +0.04]\) metres. Suppose the projection gives logits \([0.2, 0.5, 2.1, 0.3]\). Argmax selects bin index 2, so the raw normalized action is \(+0.01\). Denormalize with \(\mu_x = 0.0\), \(\sigma_x = 0.02\): \(\hat{a}_x = 0.0 + 0.02 \times 0.5 = 0.01\) m (here the bin centre is already in normalized units, so \(\hat{a}_x = 0.01\) m forward). Workspace limit check: \(0.01\) m is well inside the \(\pm 0.05\) m per-step bound, so no clamp, violation flag stays 0. The controller receives "move end-effector +1 cm in x" and queues it for the 500 Hz servo loop. Change the logits to \([0.2, 0.5, 0.4, 2.1]\) and argmax jumps to bin 3 (\(+0.04\) m), a 4x larger step from the same scene: this is exactly the quantization coarseness that compounds into the 5 mm grasp errors described above.
Take a concrete case. RT-2 receives "pick up the green block" alongside a 300x300 RGB frame of a tabletop holding three blocks. The backbone tokenizes the image into roughly 256 patch tokens and the instruction into about 10, then the transformer processes all 266 together. The action head projects the final hidden state onto 256 discrete bins per joint dimension, emitting an 8-dimensional action (6 arm joints, gripper width, vertical motion) at 3 Hz. The physical robot maps each bin index back to a joint-angle delta and feeds the command to a low-level Proportional-Derivative (PD) controller at 500 Hz. Image in, token stream out, bin-to-delta mapping, joint servo: four transitions, four separate failure points.
When deploying a pre-trained VLA on hardware that differs from the training fleet, always re-compute per-dimension action statistics (mean and standard deviation) from your own demonstration data and pass them to the normalization layer before fine-tuning. OpenVLA exposes this as --action_norm_stats at the fine-tuning entry point; skipping it causes the action head to output values in the wrong physical range even when task accuracy on held-out prompts looks correct. A quick sanity check: log the raw unnormalized action outputs for the first ten timesteps and verify they fall within the robot's joint-angle limits before running any hardware rollout.
Real-World Application: warehouse and household manipulation
Physical Intelligence's pi-zero (pi0) drives commercial robots through long-horizon tasks like folding laundry and assembling cardboard boxes by pairing a VLM backbone with a flow-matching action head (a continuous-output cousin of diffusion that learns to transform noise into an action trajectory along a smooth path rather than a fixed number of denoising steps), exactly the continuous-mode route from step 6 of the algorithm. The same interface contract lets one policy run across multiple robot embodiments, because only the action head and normalization statistics change per platform while the language-grounding front end stays shared.
The action head is a small learned module attached to the transformer's output. In discrete-token systems like RT-2, it is a linear projection onto a vocabulary of discretized action bins, one per action dimension, and the output is decoded by argmax at inference time. In continuous systems like pi-zero or RDT, it is a denoising network (diffusion or flow-matching) conditioned on the transformer's final hidden state: the network iteratively refines a noise vector into a smooth trajectory over 10 to 50 denoising steps. The choice between these two routes determines inference latency, action smoothness, and how the policy handles multi-modal distributions (where the same scene admits more than one valid motion).
The Interface Contract
Once you accept that the action head is a new output contract rather than a cosmetic swap, the next question is what exactly that contract has to specify, and the cleanest way to see it is by analogy.
Think of a sous-chef reading a recipe card (the language instruction), glancing at the ingredients on the counter (the visual observation), and then writing the next five knife strokes on a notepad before lifting the blade (the action chunk). The head chef executing those strokes works at a completely different speed than the recipe-reading step. The notepad is the interface contract: it decouples the slow, thoughtful planning cycle from the fast, precise execution cycle so neither side has to wait on the other. Remove the notepad and the chef either freezes mid-cut waiting for the next instruction, or rushes the reading and misses a critical detail.
The most useful way to read any VLA paper is to ask four interface questions. What observations enter the model? What robot state is exposed? What action space exits the model? What controller consumes those actions? This contract is the bridge back to Chapter 2 on action representations and Chapter 7 on controllers versus policies.
Code Fragment 1 makes the VLA interface concrete with typed containers. It does not run a neural policy, it shows the contract that every neural policy must satisfy. A dataclass, a Python construct that defines a lightweight typed record with auto-generated field access and no custom behavior beyond storage, keeps the observation and action fields explicit rather than buried in an untyped dictionary.
# Minimal VLA interface: image features, instruction text, and robot state enter together.
# The policy returns an action chunk, not a single ungrounded language answer.
from dataclasses import dataclass, asdict
import numpy as np
@dataclass
class VLAObservation:
image_embedding: np.ndarray
instruction: str
joint_state: np.ndarray
def as_row(self) -> dict[str, object]:
return asdict(self)
@dataclass
class ActionChunk:
delta_xyz: np.ndarray
gripper_open: np.ndarray
obs = VLAObservation(
image_embedding=np.array([0.12, 0.88, 0.41]),
instruction="pick up the red block",
joint_state=np.array([0.0, 0.4, -0.2, 0.1]),
)
chunk = ActionChunk(
delta_xyz=np.array([[0.02, 0.00, -0.01], [0.01, 0.01, -0.02]]),
gripper_open=np.array([1.0, 0.0]),
)
print(obs.instruction)
print(chunk.delta_xyz.shape)pick up the red block (2, 3)
VLAObservation and ActionChunk dataclasses define the minimal typed contract this section argues every VLA must satisfy: one image embedding, one instruction string, one joint-state vector in, a two-step delta_xyz action chunk with matching gripper commands out.The hand-built interface above is about 25 lines. With LeRobot or OpenVLA tooling, the same contract is mostly declared through dataset features and policy configuration in a few lines, while the library handles image transforms, action normalization, batching, checkpoint loading, and device placement. Keep the manual version for debugging because it names every boundary that can fail.
# LeRobot shortcut: inspect the observation and action schema before training.
# The dataset object exposes cameras, robot state, language task, and action chunks.
from lerobot.common.datasets.lerobot_dataset import LeRobotDataset
dataset = LeRobotDataset("lerobot/aloha_static_coffee")
print(dataset.features.keys())
print(dataset.meta.info.get("fps"))lerobot/aloha_static_coffee and printing its feature keys and recorded frame rate shows how the LeRobot dataset object exposes the same camera, robot-state, and action schema that Code Fragment 1 defined by hand, without writing custom loader code.Figure 34.1 should be read as the minimal VLA interface: instruction, visual observation, proprioception, action representation, and rollout evidence must all be named before behavior is interpreted.
Review and Consolidation
Curriculum, depth, and self-containment. The VLM to VLA shift is a change in output contract. The model stops producing descriptions and starts producing robot actions that must satisfy timing and safety constraints. For From VLMs to VLAs: the core idea, the practical reading is to pin down the interface, assumptions, concrete example, and failure mode before comparing methods.
Production and evaluation contract. A VLA is a policy with language conditioning, not a captioner with a gripper. For From VLMs to VLAs: the core idea, treat the diagram, code, table, exercise, warning, and references as one evidence packet: boundary, artifact, tool choice, transfer check, failure mode, and source grounding.
Before accepting a From VLMs to VLAs: the core idea result, name the loop variable that changed, the tool that makes it reproducible, the failure that would fool the metric, and the source that backs the claim.
Write an evidence row for one instruction-conditioned rollout: camera stream, language prompt, robot state, action head, success metric, latency, and the failure label that explains the first bad action.
Failure Modes
Naming every element of that interface contract is necessary but not sufficient, because the contract can be fully specified and the policy can still break where the physical boundaries bite. A VLA can fail even when its language understanding looks strong. RT-2 on a Kuka iiwa arm reaches roughly 62% success on seen pick-and-place layouts in the reported ablations. In practice, shifting the camera by as little as 15 cm from its training position has been observed to drop success below 30% in similar setups, typically because the PaLI-X backbone spreads attention over patch tokens that no longer match the expected table region. On a Franka Panda, discrete-token policies discretized into 256 bins per joint dimension can add up to roughly 5 mm of positional quantization error per step; the exact figure depends on the workspace range each bin spans. At a 3 Hz inference rate that error compounds across the 10 to 15 timesteps of a typical grasp, which in the worst case is enough to miss a 2 cm diameter peg. Proprioceptive dropout is a separate failure path. If the robot-state vector \(r_t\) lags the image frame by even 20 ms, the action head's joint-angle deltas may reference a pose the arm has already left. The low-level PD controller at 500 Hz would then chase a command that is kinematically inconsistent with the current servo readings. These are not minor implementation details. They are the reasons VLA evaluation must be closed-loop and robot-aware, with latency, viewpoint shift, and sensor synchronization treated as first-class experimental variables rather than deployment afterthoughts.
Before fine-tuning a VLA, write a one-page interface card: camera names and rates, proprioceptive fields, action dimensions, action frequency, controller type, safety limits, dataset license, and the exact success metric. This card prevents a common mistake: training a powerful model against a vague action contract.
Expected output: From VLMs to VLAs: the core idea should leave a reproducible VLA evidence trace with checkpoint, action representation, robot interface, metric, and failure label.
A reliable interface appears twice: once in the system diagram and once in the replay logs. If those two views disagree, the policy contract is still too vague.
For a tabletop pick task, name the image inputs, the robot-state vector, the language instruction, the action dimension, and the controller that executes the output. If any answer is unknown, the VLA is not yet a buildable system.
Active 2024-2026 directions:
1. Scalable cross-embodiment pretraining. The question is no longer whether web-scale data helps, but how to build shared representations across radically different embodiments (bipeds, hands, mobile manipulators). Physical Intelligence's pi0.5 (2025) and the Crossformer project (Berkeley, 2024) show that a single policy backbone can transfer across 20+ robot morphologies when embodiment tokens and action-space normalization are co-designed. The open problem is whether a truly universal action representation exists or whether each embodiment class will always need a specialized head.
2. Language-conditioned world models as VLA scaffolding. Rather than predicting actions directly, several 2024-2025 systems (UniSim from Stanford, IRASim from BAAI) learn a video-prediction world model conditioned on language and robot state, then derive actions by planning inside that model. This separates semantic grounding from motor planning and allows the policy to "imagine" the outcome of an action before committing. The open question is how to keep world-model rollouts physically consistent at the sub-second timescales robot control requires.
3. Test-time compute scaling for VLA inference. Inspired by chain-of-thought reasoning in LLMs, work from Google DeepMind (SayCan follow-ons, 2024) and CMU (ReKep, 2024) investigates whether allocating more inference-time compute, via repeated sampling, reranking, or tree search over candidate action chunks, closes the gap between a large offline-trained VLA and a smaller model with an expensive online planner. Scaling laws for action quality versus inference budget are still unmeasured.
Open problem: Current VLA evaluations conflate semantic generalization (does the model understand the instruction?) with motor generalization (does the resulting trajectory succeed on a new robot or layout?). There is no agreed decomposition benchmark that isolates these two axes. Designing a controlled evaluation suite, paired demonstrations with matched instructions but varied embodiments and camera viewpoints, would let the field measure how much of a VLA's failure comes from language grounding versus action head generalization, and would directly inform whether cross-embodiment pretraining or better action heads should be prioritized.
A VLA is best understood as a policy with a multimodal front end and an action-generating back end. The model name matters less than the observation-action contract it satisfies.
Choose one robot task and write its VLA interface card. Include observation fields, action fields, control rate, success metric, and two failure modes that a static VLM would miss.
Project Ideas
Beginner (weekend): Build a language-conditioned pick-and-place agent in PyBullet using a pretrained CLIP model as the vision encoder and a small MLP action head trained on 200 scripted demonstrations; the key challenge is aligning CLIP's image embedding with the robot's joint-angle action space without access to large-scale robot data. Intermediate (1 to 2 weeks): Fine-tune OpenVLA on a custom tabletop sorting task using LeRobot's dataset tooling and a low-cost SO-100 arm, then measure how action normalization statistics from your own demonstrations affect success rate compared to the default Open X-Embodiment statistics; the key challenge is collecting enough demonstrations (roughly 50 to 100 episodes) to shift the action distribution without catastrophic forgetting of the pretrained language grounding. Intermediate (1 to 2 weeks): Reproduce the VLM-to-VLA gap quantitatively in Isaac Lab by running a prompted GPT-4V baseline that outputs text waypoints and comparing its success rate against a small RT-1-style transformer policy trained on the same task; the key challenge is writing the waypoint-to-joint-command bridge that converts the VLM's text output into servo commands so the comparison is fair.
Lab: feel the action contract with a pretrained OpenVLA
Goal: see how the same instruction maps to different action chunks, and confirm that action normalization statistics, not the prompt, control whether outputs land in the robot's physical range. Tools: Python with transformers, the openvla/openvla-7b checkpoint from Hugging Face (a single GPU with 16 GB or Colab T4 is enough for one forward pass), and a handful of sample tabletop images (any RGB photo of objects on a table works). Steps (15 to 30 min): load the model and processor, feed one image plus the prompt "pick up the red object", and print the predicted 7-dim action vector. What to vary: (1) swap the prompt to "pick up the blue object" on the same image and compare the action deltas; (2) pass a deliberately wrong unnorm_key (a different dataset's normalization stats) and observe the action magnitudes change; (3) shift or crop the image to mimic a 15 cm camera move. What to observe: which dimensions react to the language change versus the viewpoint change, and how the unnorm key alone can push outputs outside plausible joint-delta ranges, the empirical version of the normalization warning earlier in this section.
What's Next?
Section 34.2 follows the historical path from RT-1 to RT-2 and RT-X, where action tokenization and cross-embodiment data became central ideas.
Kim et al. (2024). "OpenVLA: An Open-Source Vision-Language-Action Model." arXiv.
OpenVLA connects open VLM backbones to robot action generation and provides a practical codebase for fine-tuning. Practitioners should read it alongside the GitHub repository before adapting an open VLA to a new robot.
RT-2 made the action-as-language move explicit by fine-tuning the PaLI-X and PaLM-E backbones to emit robot actions as discretized tokens, co-trained on web VQA data and 130k RT-1 robot episodes. Read it for the co-training recipe that yields 62% success on novel objects, and for the hard limits of transferring web semantics into motor control on the Google fleet's mobile manipulators.
This paper introduced the cross-institution robot data mixture and RT-X models. It is essential for understanding why embodiment metadata, action normalization, and dataset mixture design matter.
Brohan et al. (2022). "RT-1: Robotics Transformer for Real-World Control at Scale." arXiv.
RT-1 showed that a transformer policy trained on large real robot data could produce discretized low-level robot actions from images and instructions. It is the starting point for the chapter lineage and useful for readers who want the engineering details behind large-scale robot data collection.
Hugging Face. "LeRobot." GitHub.
LeRobot is the practical open-source toolkit used here for datasets, policy training, evaluation, and low-cost robot workflows. Engineers should start here before writing custom data loaders or training loops.