"An open VLA checkpoint is a starting artifact, not a deployment claim."
A Grounded AI Agent
This section assumes familiarity with the VLM-to-VLA framing from section 34.1 and with the Open X-Embodiment dataset lineage from section 34.2. The action-head trade-off introduced here (diffusion versus regression) is examined in depth in section 34.4, which covers RDT-1B and the pi-zero family. The discrete-versus-continuous action representation question is treated formally in section 34.5 alongside the FAST tokenizer.
Discretized action tokens are convenient for transformer training, but the robot still executes metric motion, gripper commands, and timing. Always preserve the conversion back to physical units in the evaluation artifact.
A robotics team downloads Octo's weights on a Tuesday, registers their wrist camera, normalizes their gripper actions, and by Thursday they have a fine-tuned pick-and-place policy running on hardware. That speed was unthinkable two years ago. Octo and OpenVLA matter right now because open weights plus open data pipelines collapse the gap between published VLA research and working robots in real labs. You will trace how each model ingests observations, represents actions, and adapts to new hardware, giving you the vocabulary to evaluate, fine-tune, and debug generalist policies on your own setup.
Why Open Generalist Policies Matter
Closed demonstrations can inspire a field, but open policies teach it how to deploy. Before Octo and OpenVLA, adapting a pretrained manipulation policy to a new robot meant three painful steps. You reverse-engineered proprietary action representations, guessed normalization statistics, and reproduced dataset filtering logic from paper descriptions alone. Open weights change that. A lab with a Franka Panda, a wrist-mounted Intel RealSense, and 100 teleoperated episodes can audit the exact dataset_statistics.json the pretraining used. The same lab can confirm that 6-DoF end-effector deltas are normalized per dimension to the Open X-Embodiment distribution, then run a reproducible fine-tune in under eight hours on a single A100. The failure surfaces are inspectable too, which matters because the most common deployment failures are not model quality issues but interface mismatches: wrong camera slot assignment, un-inverted gripper convention, or action dimensions whose pretraining range does not cover the physical robot's workspace.
Octo is a transformer-based diffusion policy pretrained on Open X-Embodiment trajectories spanning more than 70 robot embodiments, with named observation slots ("primary" and "wrist" cameras) and an action head that outputs continuous 7-Degrees-of-Freedom (DoF) end-effector deltas via a denoising process (a diffusion policy predicts an action by starting from random noise and iteratively refining it toward a plausible action, conditioned on the current observation; section 34.4 covers this mechanism in depth, but the short version is enough to follow the comparison below). OpenVLA uses an Internet-pretrained 7B Vision-Language Model (VLM) backbone (Prismatic, a lightweight VLM architecture that fuses a frozen vision encoder with a language model backbone, built here on SigLIP, an image-text encoder trained with a sigmoid contrastive loss) and discretizes end-effector actions into 256 bins per dimension. Fine-tuning teaches the language model to predict action tokens rather than words. In a hardware integration review, compare them on four embodiment-specific axes: which physical cameras map to which named input slots, whether the action space is continuous deltas or discretized bins, what pretraining normalization statistics apply to your robot's gripper range, and how inference latency at the control-loop frequency (commonly 5 to 10 Hz for manipulation) fits within the robot controller's timing budget.
An open VLA is valuable not only because you can run it. You can inspect the dataset interface, reproduce fine-tuning, measure latency, change the action head, and discover where the policy breaks.
A Practical Selection Guide
| Question | Octo-style answer | OpenVLA-style answer |
|---|---|---|
| Main strength | Robot-data generalist initialization | VLM semantics plus robot action fine-tuning |
| Best early use | Fine-tune on a robot setup with related data | Study language-conditioned manipulation with open tooling |
| Primary risk | Limited semantic transfer outside robot data | Strong semantics without reliable physical grounding |
| Debug handle | Observation and action adapters | Prompt, tokenizer, and action decoding path |
The table above is a starting point, not a leaderboard. Use it to choose experiments. Do not use it to declare a universal winner because the answer depends on robot, task, dataset, and evaluation protocol.
Getting the Action Units Right
Whichever model that experiment-driven comparison points you toward, the first thing it will demand of your hardware is the same: getting the action units right. Action normalization matters physically because a robot's gripper may travel 2 cm while the pretraining dataset spans 20 cm wrist motions; without rescaling, the policy's learned weight magnitudes map to the wrong physical scale and the arm either stalls or overshoots on every step. A mismatch of one order of magnitude in the action range is enough to make a policy unsafe on hardware even if it looks correct in simulation. A policy that moves the right distance in the wrong units is not a calibrated policy; it is a hazard wearing good intentions. The normalization step translates your robot's metric workspace into the statistical units the pretrained weights expect, preserving the learned motion priors while adapting their physical magnitude.
Mechanically, the transform is per-dimension z-scoring, where each raw value is rescaled by its own mean and standard deviation so the result has zero mean and unit variance: for each of the \(d\) action dimensions, subtract the pretraining mean \(\mu_a^{(i)}\) and divide by the pretraining standard deviation \(\sigma_a^{(i)}\), yielding \(\hat{a}_t^{(i)} = (a_t^{(i)} - \mu_a^{(i)}) / \sigma_a^{(i)}\). These statistics come from the released dataset_statistics.json. At inference time the inversion \(a_t^{(i)} = \hat{a}_t^{(i)} \cdot \sigma_a^{(i)} + \mu_a^{(i)}\) converts network outputs back to metric commands before they reach the robot controller.
Think of action normalization like converting a recipe from one cook's handwritten notes to a standardized kitchen scale. The original cook wrote "a handful of salt" because their hand and their salt shaker produced a consistent amount, but your hand is a different size. Z-scoring translates "a handful" into grams: it measures what the original cook's typical handful weighed (the pretraining mean) and how much it varied (the standard deviation), then tells you exactly how many grams your robot needs to pour. Without that conversion, you are following a recipe in the wrong units and every dish comes out oversalted or bland, even though the instructions are technically correct.
Consider a specific case: a team wants to teach a UR5 arm to pick up a coffee cup using a wrist-mounted RGB camera and a parallel-jaw gripper. They have 150 teleoperated episodes, each about 30 seconds at 10 Hz, yielding roughly 45,000 transition tuples. With Octo, they load the Open X-Embodiment checkpoint, register their single camera as the "primary" observation slot, normalize the 6-DoF end-effector delta actions to the dataset mean and standard deviation, and run a fine-tune for 20,000 gradient steps on a single A100. At evaluation, they hold out 20 episodes and measure task success rate.
With OpenVLA they swap in the 7B VLM backbone, format the task as the prompt "pick up the coffee cup," and fine-tune the action head for the same episode count. The two policies fail differently at test time. Octo's diffusion head produces smooth trajectories but struggles with novel cup colors absent from the pretraining mix. OpenVLA's language grounding picks the correct cup even beside a distractor, but pays for it: roughly 150 ms per step against 30 ms for the smaller Octo head. That gap is a latency-versus-generalization trade-off, not a quality ranking. Over a 30-second trial at 10 Hz, the 120 ms per-step deficit accumulates to 36 seconds of pure inference delay, so the OpenVLA robot runs in slow motion and misses a sliding object the Octo policy catches cleanly. The 7B backbone carries roughly 50 times more parameters than a compact Octo action head; fine-tuning wall-clock stays comparable at 150 episodes, but inference cost is paid on every control cycle.
Checkpoint
So far: action normalization converts a robot's raw motion units into the pretrained model's statistical scale via z-scoring and its inversion, and the same UR5 coffee-cup case study shows that Octo and OpenVLA trade off inference latency against language-grounded generalization rather than one simply outperforming the other.
Algorithm: Open Generalist Policy Adaptation
Input: pretrained checkpoint \(\theta_0\) (Octo or OpenVLA), robot dataset \(\mathcal{D} = \{(o_t, a_t)\}\) of \(N\) episodes, language task description \(\ell\), action normalization statistics \(\mu_a, \sigma_a\) from pretraining
Output: adapted policy \(\pi_\theta\) ready for closed-loop rollout on the target robot
- Inspect the pretraining observation schema and map each robot camera to the model's named input slots (for example, "primary" and "wrist" in Octo); verify image resolution and channel order match.
- Load \(\mu_a, \sigma_a\) from the released
dataset_statistics.json; compute per-dimension mean and standard deviation from \(\mathcal{D}\); apply the transform \(\hat{a}_t = (a_t - \mu_a) / \sigma_a\) to every action in the dataset. - Verify that \(\hat{a}_t \in [-2, 2]^d\) for all \(t\); flag and clip any outlier dimensions before proceeding.
- Encode the task description \(\ell\) through the model's language tokenizer; confirm token length fits within the model's context window.
- Initialize policy parameters \(\theta \leftarrow \theta_0\) and freeze the vision encoder if GPU memory is limited (freeze all layers except the action head and cross-attention adapters for the smallest fine-tune footprint).
- Run supervised fine-tuning with learning rate \(\alpha\) (typically \(10^{-4}\) to \(10^{-5}\)): \(\theta \leftarrow \theta - \alpha \nabla_\theta \mathcal{L}(\pi_\theta, \mathcal{D})\), where \(\mathcal{L}\) is the action prediction loss (MSE, mean squared error, the average squared difference between predicted and true actions, for regression heads, diffusion score loss for Octo's head).
- After every \(K\) gradient steps, run a short closed-loop rollout on the robot and record the success rate on the held-out evaluation set \(\mathcal{D}_\text{eval}\).
- If success rate plateaus below target, inspect the failure videos; add targeted episodes covering the specific failure modes (novel backgrounds, workspace-edge placements, partial grasps) and resume fine-tuning.
- Invert the action normalization at inference time: \(a_t = \hat{a}_t \cdot \sigma_a + \mu_a\); confirm that gripper open/close convention matches the physical robot (one-bit sign errors are the most common deployment failure).
- Log checkpoint hash, normalization statistics, fine-tuning episode count, evaluation seed, and success rate as one artifact before deployment.
Step-Through: Action Normalization and Inversion
Trace the per-dimension z-scoring transform with one concrete action dimension: the x-axis end-effector delta. Suppose the released dataset_statistics.json gives pretraining mean \(\mu_a^{(x)} = 0.004\) m and standard deviation \(\sigma_a^{(x)} = 0.018\) m. Your robot teleoperates a step of \(a_t^{(x)} = 0.022\) m (a 22 mm forward nudge). Normalize: \(\hat{a}_t^{(x)} = (0.022 - 0.004) / 0.018 = 0.018 / 0.018 = 1.0\). That value sits comfortably inside the \([-2, 2]\) sanity window, so it passes the outlier check. Now suppose the network, at inference, outputs \(\hat{a}_t^{(x)} = -0.5\). Invert: \(a_t^{(x)} = (-0.5)(0.018) + 0.004 = -0.009 + 0.004 = -0.005\) m, that is, a 5 mm backward command sent to the controller. If you had skipped normalization and fed the raw 0.022 directly to a head expecting unit-scale inputs, the loss would still fall but the rollout would treat 0.022 as if it were 0.022 standard deviations: a near-zero motion, and the arm would stall. One forgotten division by \(\sigma\) turns a 22 mm reach into a frozen gripper.
When fine-tuning Octo, retrieve the pretraining normalization statistics from the released dataset_statistics.json file and compute your robot's per-dimension mean and standard deviation from your own episodes before training. Octo's action head expects inputs already scaled to the pretraining distribution, so mixing your raw gripper range with the dataset's normalized deltas produces silent numerical garbage: the loss decreases normally but rollout actions are wildly out of range. The safest check is to print the min and max of your normalized actions after applying the transform and confirm they fall roughly in the interval [-2, 2] before the first gradient step.
Manual fine-tuning scripts quickly grow past 100 lines once they include video loading, normalization, episode slicing, and checkpointing. LeRobot and OpenVLA tooling reduce that to configuration plus one training command, while handling dataset adapters, transforms, logging, and model loading internally.
# Practical route: use a maintained training entry point instead of custom loaders.
# Check the current repository docs before running because model names evolve.
python -m lerobot.scripts.train configs/smolvla_aloha_static_coffee.yamlOnce a maintained entry point like that is handling the training loop, your attention shifts from writing code to versioning the artifacts that flow through it. Figure 34.3 should be read as an adaptation pipeline: checkpoint, tokenizer or encoder, robot interface, fine-tuning data, calibration, and rollout logs each require their own version record.
An open VLA succeeds or fails at the seams between these four boxes, not inside them. Camera-to-slot mapping enters at Vision, action normalization and gripper convention live at the Action Head, and the feedback arrow shows why deployment is a closed loop: rollout failures must flow back as targeted fine-tuning data, not as one-shot inference. This is the same diagram introduced as Figure 34.1.
Review and Consolidation
Curriculum, depth, and self-containment. Octo and OpenVLA represent two open routes: generalist diffusion-policy initialization and open VLM-based action generation. For Open generalist policies: Octo, OpenVLA, the practical reading is to pin down the interface, assumptions, concrete example, and failure mode before comparing methods.
Production and evaluation contract. Open weights matter because they let readers inspect data adapters, action heads, and fine-tuning recipes. For Open generalist policies: Octo, OpenVLA, treat the diagram, code, table, exercise, warning, and references as one evidence packet: boundary, artifact, tool choice, transfer check, failure mode, and source grounding.
Before accepting a Open generalist policies: Octo, OpenVLA result, name the loop variable that changed, the tool that makes it reproducible, the failure that would fool the metric, and the source that backs the claim.
Write an evidence row for one open-policy adaptation: base checkpoint, robot observations, action adapter, fine-tuning episodes, evaluation seed, and the failure mode that blocks deployment.
When Direct Execution Is Impractical
Open VLA models are still heavier than most compact examples in this book. On a small laptop or a 6 GB GPU, readers should start with dataset inspection, schema validation, and a tiny policy head before attempting full fine-tuning. The goal is to understand the data contract first, then scale compute when the contract is clean.
Start with a single task and 20 held-out episodes. Verify that the model consumes the correct cameras, that action normalization inverts cleanly, that inference latency fits the control loop, and that failure videos are saved. Only then increase dataset size or model scale.
Real-World Application: Warehouse Bin Picking
Physical Intelligence and academic labs building on the OpenVLA codebase fine-tune open generalist checkpoints for warehouse pick-and-place, where a single arm must grasp thousands of distinct SKUs it never saw during pretraining. The open weights let integrators audit the exact action normalization and camera-slot mapping before deployment, then close the sim-to-real gap with a few hundred on-robot episodes per cell. This auditability is typically a factor in why open VLAs have moved from demo videos to production pilots faster than comparable closed manipulation policies, though direct pilot-count comparisons across vendors are not publicly tracked.
Both Octo and OpenVLA can degrade sharply when the deployment robot's camera viewpoint, lighting, or action space differs from the pretraining distribution. In practice, teams report a policy that achieves 80% success on the training embodiment can drop below 20% on a structurally similar robot with a slightly different wrist-camera angle or a gripper whose open/close range was not renormalized. Fine-tuning on even 50 to 100 on-robot episodes usually recovers performance, but only if the fine-tuning data covers the specific failure modes: novel backgrounds, object placements near workspace edges, and recovery from partial grasps are the most common gaps.
A common assumption is that the label "generalist" means Octo or OpenVLA will work zero-shot on any new robot without adaptation, because the models were pretrained across dozens of embodiments. This is wrong in embodied AI: "generalist" refers to breadth of pretraining coverage, not to zero-shot transferability. A new robot's camera placement, gripper range, and action convention almost certainly differ from every embodiment in the pretraining mix, so the policy's learned priors map to the wrong physical interface and produce unsafe or incoherent behavior. The correct mental model is that open generalist policies provide a strong initialization, not a finished product; targeted fine-tuning on even 50 to 150 on-robot episodes is required to bridge the gap between the pretraining distribution and a specific deployment setup.
If a policy fails, first check whether the gripper convention is inverted. Many dramatic robot failures reduce to one bit meaning "open" in one dataset and "closed" in another.
Inspecting and Scaling Up
You inherit a robot dataset with three cameras and a 7-dimensional action vector. Which parts of Octo or OpenVLA adaptation would you inspect before training?
Efficient open-weight VLAs for edge deployment. SmolVLA (Hugging Face, 2025) demonstrated that sub-1B parameter VLAs trained on community-contributed LeRobot datasets can match larger checkpoints on standard manipulation benchmarks. The 2024-2026 direction is aggressive compression: quantization-aware fine-tuning, layer pruning, and distillation from 7B teachers into 100M-300M student policies that run inference within a 10 Hz control loop on an NVIDIA Jetson or Apple M-series chip without a cloud GPU dependency.
Parameter-efficient adaptation methods. Full fine-tuning of a 7B VLA backbone on 100 robot episodes risks catastrophic forgetting of the VLM's language grounding. The 2024 OpenVLA-OFT paper (Kim et al., 2024, arXiv:2411.18270) showed that parallel adaptation layers with action chunking and full fine-tuning of only the newly added action head outperforms LoRA (Low-Rank Adaptation, a technique that fine-tunes a small pair of low-rank matrices instead of the full weight matrix) alone on dexterous tasks. Active research explores which transformer layers store robot-relevant vs. language-only knowledge so that targeted freezing can preserve both capabilities.
Data-efficient few-shot generalization. The primary bottleneck for open generalist policies is not model capacity but demonstration data: getting from 0 to 80% success on a new task still requires 50 to 200 teleoperated episodes. Work from the Berkeley Robot Learning Lab (2025) on retrieval-augmented fine-tuning retrieves the most similar trajectories from the Open X-Embodiment pool at training time, reducing the required new-robot episodes by roughly half on tabletop tasks.
Open problem for PhD students. Current evaluation of open VLAs collapses performance into a single task-success number measured under one lighting condition and one object placement distribution. There is no accepted protocol for measuring the degradation curve as viewpoint, object color, or workspace boundary shifts away from the fine-tuning distribution. Designing a standardized covariate-shift evaluation suite, analogous to ImageNet-C for image classifiers, that can be run on both simulated and real robots without additional human teleoperation, is an open and tractable dissertation problem.
Open generalist policies turn VLA research into an engineering workflow: inspect the schema, adapt the interface, fine-tune carefully, and evaluate on held-out closed-loop behavior.
Pick Octo, OpenVLA, or SmolVLA. Write a fine-tuning plan for a new tabletop task, including required data fields, compute assumptions, held-out tests, and the first failure video you would inspect.
Lab: Probe the Action-Normalization Contract
Goal: see firsthand how the released pretraining statistics shape the action distribution your fine-tune actually sees, and why a unit mismatch silently breaks rollouts. Tools needed: Python with NumPy and Matplotlib, plus one LeRobot dataset (for example the push-T set pulled via lerobot.common.datasets.lerobot_dataset.LeRobotDataset); no GPU required. Steps: load one episode, extract the raw action array, then load the dataset mean and standard deviation from the dataset metadata (or the model's dataset_statistics.json). Apply \(\hat{a} = (a - \mu) / \sigma\) per dimension and plot a histogram of every dimension before and after. What to vary: deliberately corrupt one dimension's \(\sigma\) by 10x and by 0.1x, then re-normalize. What to observe: on the clean transform, normalized values should cluster inside \([-2, 2]\); under the corrupted \(\sigma\) they spill far outside, exactly the silent failure that makes a fine-tuned policy stall or overshoot on hardware. Record the fraction of timesteps outside \([-2, 2]\) as a one-number health check you can reuse on any robot dataset.
Project Ideas
Beginner (weekend): Fine-tune SmolVLA on the LeRobot community push-T dataset using the LeRobot training script and evaluate success rate on 20 held-out episodes in the Gymnasium push-T environment; the key challenge is correctly mapping the dataset action normalization statistics so that rollout actions stay within the environment's valid range. Intermediate (1 to 2 weeks): Adapt Octo to a simulated UR5 arm in MuJoCo or PyBullet by collecting 100 teleoperated pick-and-place episodes via a ROS2 teleoperation node, registering the wrist camera as the "wrist" observation slot, and comparing fine-tuned task success against the zero-shot baseline; the key challenge is debugging the gripper convention mismatch between the pretraining dataset and the simulated controller before training begins. Advanced (2 to 3 weeks): Build a language-conditioned bin-sorting policy in Isaac Lab by fine-tuning OpenVLA on synthetic demonstration data generated with Isaac Lab's randomized object placement, then deploy the same checkpoint on a physical robot arm via a ROS2 action server and measure the sim-to-real success gap; the key challenge is closing the visual domain gap between Isaac Lab's rendered camera images and the real wrist-camera feed without additional real-robot data collection.
What's Next?
Section 34.4 explains why several frontier systems use diffusion or flow heads instead of plain action tokens.
SmolVLA is a compact open VLA designed to run on more accessible hardware and fine-tune on LeRobot datasets. It is the best fit for the chapter hands-on lab because it lowers the barrier to experimentation.
Octo Model Team et al. (2024). "Octo: An Open-Source Generalist Robot Policy." arXiv.
Octo is a transformer-based diffusion policy pretrained on Open X-Embodiment trajectories and designed for flexible fine-tuning. It is the clearest open reference for generalist policy initialization before the Internet-pretrained VLA wave.
Kim et al. (2024). "OpenVLA: An Open-Source Vision-Language-Action Model." arXiv.
OpenVLA connects open VLM backbones to robot action generation and provides a practical codebase for fine-tuning. Practitioners should read it alongside the GitHub repository before adapting an open VLA to a new robot.
This paper introduced the cross-institution robot data mixture and RT-X models. It is essential for understanding why embodiment metadata, action normalization, and dataset mixture design matter.
OpenVLA Project. "OpenVLA GitHub Repository." GitHub.
The repository contains training and fine-tuning code for OpenVLA-style policies. It is the implementation reference when the chapter discusses open tooling rather than closed vendor demos.
Hugging Face. "LeRobot." GitHub.
LeRobot is the practical open-source toolkit used here for datasets, policy training, evaluation, and low-cost robot workflows. Engineers should start here before writing custom data loaders or training loops.