Section 22.7: Choosing an action representation: a decision guide

"The action representation you pick is a bet on what the task will punish you for getting wrong."

A Cautious Policy Interface

Two engineers feed the exact same bimanual demonstrations into their policies; one ships a robot that inserts a USB plug on the first try, the other ships a robot that lunges at the gap between two valid grasps and faults the controller, and the only thing they did differently was pick an action representation. By this point in the chapter you have seen five different ways to parameterize a robot's output: a single continuous action, an ACT chunk (ACT stands for Action Chunking with Transformers), a diffusion-sampled chunk, a flow-matched chunk, and a discrete behavior token (VQ-BeT, short for Vector-Quantized Behavior Transformer, which snaps each action to the nearest entry in a learned codebook). Each was motivated by a real limitation of the one before it, and each grew out of the broader imitation learning setting where a policy must reproduce demonstrated behavior. The question practitioners face is not which method is theoretically superior but which one fits a specific robot, task, and deployment constraint. A screwing task at 20 Hz with a low-cost arm has different binding constraints than a warehouse picking task at 5 Hz on a commercial manipulator. This section gives you a structured way to read those constraints and map them to an action family before you commit to a training run.

The question this guide answers

This section develops the technical contract for choosing an action representation: a decision guide into a usable mental model. First we define the object of study, then we connect it to the agent loop, then we test it with a compact implementation.

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Worked Example

To see the contract turn from an abstract checklist into a concrete verdict, run it against the very task that opened this section. Take a concrete case: an ALOHA bimanual cell inserting a USB-C plug at 20 Hz, where two valid wrist approaches (left tilt and right tilt) appear in the demonstrations. The binding constraints are a 50 ms control step and submillimeter contact tolerance. The snippet below scores each action family against those two numbers so the choice is made on the spec, not on reputation.

# Per-family fit for ALOHA USB-C insertion: 20 Hz (50 ms step), bimodal grasp.
step_budget_ms = 1000 / 20          # 50 ms control loop on the ALOHA arms
fwd_ms = 5                          # one score-net forward pass on the deployment GPU
families = {
    "one_step":   {"latency_ms": fwd_ms * 1,  "handles_bimodal": False},
    "act_chunk":  {"latency_ms": fwd_ms * 1,  "handles_bimodal": False},
    "diffusion":  {"latency_ms": fwd_ms * 10, "handles_bimodal": True},
    "flow_match": {"latency_ms": fwd_ms * 5,  "handles_bimodal": True},
    "vq_bet":     {"latency_ms": fwd_ms * 1,  "handles_bimodal": True},  # but quantizes contact
}
for name, f in families.items():
    fits = f["latency_ms"] <= step_budget_ms and f["handles_bimodal"]
    print(name, f["latency_ms"], "ms", "FITS" if fits else "rejected")

Expected output: only flow matching satisfies both constraints once the precision gate eliminates VQ-BeT. Change the control frequency to 5 Hz (a 200 ms budget, typical of a warehouse pick on a UR5) and diffusion re-enters the feasible set, which is exactly the point: the binding constraint, not the model's expressiveness, decides the family.

# Score candidate action horizons using latency and task-temporal needs.
# Lower latency cost and higher intent coverage make a horizon preferable.
horizons = [1, 4, 8, 16]
control_hz = 20
for horizon in horizons:
    intent_seconds = horizon / control_hz
    latency_penalty = 0.015 * horizon
    score = intent_seconds - latency_penalty
    print(horizon, "steps", "intent_s", round(intent_seconds, 2), "score", round(score, 2))

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

The decision guide pays off only when tied to a closed-loop contract that names the observation stream, the state estimate, the action representation, the timing budget, and the evaluation artifact. Skip that contract and a model can look capable in a notebook, then fail the first time a sensor drops a frame or a controller saturates.

Separate the conceptual claim, the systems claim, and the evidence claim. A plausible mechanism, a clean interface, and a closed-loop result are different claims; the section should keep their evidence separate.

Start with a small baseline that logs inputs, outputs, units, timestamps, and termination conditions before moving to Gymnasium or PettingZoo. The library run should keep the same artifact schema, so the comparison remains a same-task evaluation.

1. Write a one-paragraph task contract with observation, action, success, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save a single result artifact containing configuration, seed, metrics, videos or traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel.

When Choosing an action representation: a decision guide fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.