Section 34.5: Action tokenization vs. continuous heads; the FAST tokenizer

"Action representation is where a language model becomes a robot controller or fails to become one."

A Grounded AI Agent

The Action Representation Problem

Two engineers train the same Vision-Language-Action model (VLA) on the same demonstrations, yet one robot grasps cleanly and the other stutters into the cup. The only difference: how each team turned continuous motion into symbols the model could predict. A VLA can only learn actions through the representation it is given, and if that representation loses timing, smoothness, gripper convention, or coordinate meaning, no amount of language understanding repairs it. This is why action tokenization, action chunking, diffusion heads, and flow heads belong in the same design conversation.

A tokenizer that loses a robot's timing is not a compression step; it is a mistranslation between intention and motion.

Figure 34.5A previews the core tradeoff this section unpacks: discrete FAST tokens reconstruct high-frequency motion with lower inference latency, while a continuous regression head produces smoother curves but cannot represent multiple valid action modes.

Four ways to represent an action

Naive tokenization bins each action dimension at each timestep. It is easy to implement, but it can produce long token sequences and visible quantization error. Action chunking predicts several future actions at once, which improves temporal consistency. Diffusion and flow heads generate continuous chunks directly. FAST keeps the token route but compresses action sequences in frequency space before tokenization.

Code Fragment 1 below shows a tiny version of per-dimension action tokenization. The example is intentionally small so the quantization error is visible.

# Discretize continuous end-effector deltas into fixed bins.
# This teaches the source of tokenization error before FAST compresses full action sequences.
import numpy as np

values = np.array([-0.041, -0.012, 0.006, 0.019, 0.044])
bins = np.linspace(-0.05, 0.05, 9)
token_ids = np.digitize(values, bins) - 1
centers = (bins[:-1] + bins[1:]) / 2
reconstructed = centers[np.clip(token_ids, 0, len(centers) - 1)]
print(token_ids.tolist())
print(np.round(reconstructed - values, 4).tolist())

[0, 3, 4, 5, 7]
[-0.0028, 0.0058, 0.0002, -0.0002, -0.0002]

If binning each timestep independently is what inflates sequence length and injects that visible quantization error, the natural fix is to compress the sequence before tokenizing it at all, which is exactly what FAST does.

FAST In Plain Language

FAST stands for Frequency-space Action Sequence Tokenization. Instead of tokenizing every raw action dimension at every timestep, it transforms an action sequence into frequency coefficients, then tokenizes the compressed representation. This is the token count cliff: a 50-step, 7-DOF (degree-of-freedom, meaning one independently controllable joint or axis) action chunk produces 350 raw tokens per chunk with naive binning, but only 16 to 32 FAST tokens after DCT (Discrete Cosine Transform) compression on smooth reaching motions. The intuition is familiar from signal processing: many smooth robot motions can be described with fewer low-frequency components than raw samples.

Why token count matters on a real robot. A robot servo loop does not wait for the policy. At 50 Hz the controller expects a new action command every 20 ms; if autoregressive decoding of 350 tokens takes 40 ms on the onboard GPU, the controller either stalls or replays a stale command, producing visible jitter or missed contact timing. Cutting the sequence to 20 tokens brings inference inside the 20 ms budget, making real-time closed-loop control possible without faster hardware.

How DCT compression achieves the reduction. A smooth reaching trajectory changes slowly across timesteps, so most signal energy concentrates in the first few cosine basis functions (low frequencies). The Discrete Cosine Transform reorganizes the action sequence into those basis coefficients ordered by frequency; truncating after the first \(K_{\text{freq}}\) coefficients discards only the near-zero high-frequency terms. The surviving coefficients are then quantized and packed into a short token sequence via Byte Pair Encoding (BPE).

# Pseudocode for a FAST-style tokenizer interface.
# Use the current openpi or FAST implementation rather than reimplementing DCT+BPE.
tokenizer = load_action_tokenizer("fast_plus")
tokens = tokenizer.encode(action_chunk, robot_metadata=metadata)
restored_chunk = tokenizer.decode(tokens, robot_metadata=metadata)

if isinstance(restored_chunk, list):
    print({"rows": len(restored_chunk), "first": restored_chunk[0] if restored_chunk else None})
elif isinstance(restored_chunk, dict):
    print({"fields": sorted(restored_chunk), "audit_ready": all(value not in (None, "") for value in restored_chunk.values())})
else:
    print({"value": restored_chunk})

The intuition, the compression pipeline, and the algorithm above cover how FAST tokenizes actions; the remaining piece the section title promises is a decision rule for choosing between FAST tokens and continuous heads on a new robot, which the next paragraph and the Decision Guide table deliver directly.

The choice between FAST tokens and continuous heads (diffusion or flow) turns on three concrete factors. First, inference latency. An autoregressive token model decodes a 50-step action chunk in a single forward pass once it predicts the tokens. A diffusion head instead runs 10 to 100 denoising steps per chunk, which adds 50 to 200 ms on a typical GPU (as of 2024; faster samplers and dedicated hardware continue to narrow this gap). At 50 Hz each servo tick is 20 ms, so a 200 ms diffusion head makes the robot execute a command computed 10 timesteps ago. At a typical reaching speed of around 0.1 m/s, the arm has by then already moved roughly 2 cm from where the policy placed it when it chose that action; slower or faster motions shift this figure accordingly. Second, multimodality: if two qualitatively different grasp strategies are equally valid for the same observation, a diffusion or flow head can represent both modes; a single token sequence collapses to one mode. Third, training data smoothness: FAST compression is most efficient when actions are smooth over the prediction horizon, because the DCT concentrates energy in low-frequency coefficients. Jerky or contact-rich motions (peg insertion, door unlatching) spread energy across many coefficients and recover fewer tokens than smooth reaching motions, partially eroding FAST's sequence-length advantage. A practical rule: if your task is smooth and your inference budget is tight, FAST tokens; if your task is multi-modal or contact-rich and you can afford the sampling cost, a diffusion or flow head.

Figure 34.5 should be read as an action-interface comparison: discrete tokens, continuous heads, chunked actions, rate limits, and inverse transforms must be audited together.

With the interface, its failure modes, and the latency stakes now pinned down, those scattered tradeoffs collapse into a single lookup table that maps each representation to the situation it fits.

Decision Guide

Expected output: Action tokenization vs. continuous heads; the FAST tokenizer should leave a reproducible VLA evidence trace with checkpoint, action representation, robot interface, metric, and failure label.