Section 34.7: Prompting and conditioning embodied policies

"A robot policy is a promise about the next second of the world."

A Grounded AI Agent

Prompting Is Runtime Conditioning

Change one word in a robot's instruction, from "pick up the red block" to "grab the red block," and its measured success rate on the exact same scene can fall from 84 percent to 61 percent, with no change to the model, the camera, or the object. As Figure 34.7A suggests, a prompt for an embodied policy is not a request for text but a setting that shapes physical motion. In a language model, a prompt can ask for a style or answer format. In a Vision-Language-Action model (VLA), a prompt can change motor behavior. That makes prompt design part of the control interface. The prompt must specify the goal, relevant object, constraints, and stop condition without asking the low-level policy to reason beyond its grounding. A word that feels synonymous to a human can feel physically distinct to a policy: it routes the arm down a different trajectory entirely.

Conditioning can also come from goal images, robot state, task embeddings, skill identifiers, or structured plans from an LLM planner. Task embeddings deserve particular attention. A robot cannot pause mid-motion to reinterpret an ambiguous instruction. A misread embedding commits the arm to a trajectory that may collide with an object or exceed joint limits before any correction arrives. A task embedding is the fixed-length vector the VLA's frozen language encoder produces from the instruction, and the action head reads this vector at every control step as a learned bias. Nearby vectors activate similar motor programs, because training demonstrations with similar motion primitives produced them (the next paragraph explains this embedding-proximity mechanism in detail). Conditioning narrows the policy distribution. A well-specified instruction can cut the demonstrations needed to learn a reliable grasp from roughly 50,000 unconditioned episodes down to around 300 language-conditioned ones, because the embedding steers every gradient update toward one cluster of relevant motions instead of averaging across all tasks. A vague instruction such as "clean this up" may belong to a task planner in Chapter 33. The VLA needs a grounded command such as "pick up the red block and place it in the blue tray."

The mechanism behind this sensitivity is embedding proximity. A VLA encodes the prompt into a language embedding (the same vector called the task embedding above) that conditions the action head. Phrases that sit close together in the training corpus produce similar embeddings, and therefore similar action distributions. "Move the red block" and "push the red block" may map to embeddings close enough to activate overlapping motor programs, while "pick up the red block" activates a grasp trajectory. So prompt wording is not merely descriptive. It is a selector over the learned action repertoire: on an OpenVLA fine-tune for a standard pick-and-place task (as of 2024), switching "pick up" to "grab" dropped task success from 84 percent to 61 percent with no other change. Synonyms that feel equivalent to a human can point to different clusters in demonstration space, which is why evaluating prompt variants before deployment is necessary rather than optional.

Prompt Patterns

If the prompt is a contract, then contracts fall into a few recurring shapes, and the table below names the four that cover most tabletop manipulation tasks.

Because wording alone can pick the wrong skill, the same conditioning mechanism that makes these deployments work also produces a small set of predictable ways prompts break down.

Building that hierarchy in practice means assigning each instruction source a fixed priority before deployment, not adjudicating conflicts at runtime. A workable ordering is: (1) hard-coded system safety constraints (never overridden), (2) the current operator command, (3) perception-derived labels used only to resolve object references, never to introduce new goals, and (4) any text or speech present in the environment, which the policy should treat as untrusted context rather than as an instruction. Concretely, this means the prompt-construction pipeline should reject or flag any candidate instruction that originates from perception or environmental text if it conflicts with the active operator command or safety constraint, rather than passing all sources into the language encoder as equally weighted text.

# Install the open robot-learning toolkit and common notebook dependencies.
# Check the LeRobot repository for the current extras before a real fine-tune.
pip install lerobot numpy pandas

# Dataset card: record the robot, sensors, action space, and task language.
# Fill the reader fields before collecting or fine-tuning any demonstrations.
from dataclasses import dataclass

@dataclass
class VLADatasetCard:
    robot: str
    cameras: list[str]
    action_space: str
    control_hz: int
    prompt_template: str
    success_metric: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

card = VLADatasetCard(
    robot="aloha_static",
    cameras=["wrist_rgb", "front_rgb"],
    action_space="7D end-effector delta plus gripper state",
    control_hz=10,
    prompt_template="pick up the {object} and place it on the {target}",
    success_metric="object center lies inside tray after release",
)
print(card)

# Prompt variants: test whether wording changes the intended task semantics.
# Keep the action goal stable while varying object names and constraints.
templates = [
    "pick up the {object} and place it on the {target}",
    "move the {object} to the {target} without touching the distractor",
    "grasp the {object}, lift it, then release it over the {target}",
]
for template in templates:
    print(template.format(object="red block", target="blue tray"))

# Evaluation panel: all policies must run on these same scenarios and seeds.
# Add perturbations that test language, perception, and control separately.
import pandas as pd

panel = pd.DataFrame([
    {"episode": 1, "object": "red block", "target": "blue tray", "lighting": "normal", "seed": 11},
    {"episode": 2, "object": "red block", "target": "blue tray", "lighting": "dim", "seed": 12},
    {"episode": 3, "object": "red cube", "target": "blue tray", "lighting": "normal", "seed": 13},
])
print(panel)

# LeRobot shortcut: inspect a dataset schema before choosing a policy class.
repo_id = "lerobot/aloha_static_coffee"
policy_options = ["act", "diffusion_policy", "openvla_adapter"]
selected = policy_options[0]
command = f"python -m lerobot.scripts.train configs/{selected}.yaml"
print({"repo_id": repo_id, "policy_options": policy_options, "initial_policy": selected, "command": command})

# Complete lab solution: dataset card, prompt variants, and evaluation panel.
# This is a planning artifact that should run before any expensive VLA fine-tune.
from dataclasses import dataclass
import pandas as pd

@dataclass
class VLADatasetCard:
    robot: str
    cameras: list[str]
    action_space: str
    control_hz: int
    prompt_template: str
    success_metric: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

card = VLADatasetCard(
    robot="ALOHA-style dual-arm tabletop robot",
    cameras=["front_rgb", "left_wrist_rgb", "right_wrist_rgb"],
    action_space="14D joint position targets plus two gripper commands",
    control_hz=20,
    prompt_template="pick up the {object} and place it on the {target}",
    success_metric="object center is inside target region after release",
)

templates = [
    "pick up the {object} and place it on the {target}",
    "move the {object} to the {target} without touching the distractor",
    "grasp the {object}, lift it, then release it over the {target}",
]
prompts = [template.format(object="red block", target="blue tray") for template in templates]

panel = pd.DataFrame([
    {"episode": 1, "object": "red block", "target": "blue tray", "lighting": "normal", "seed": 11},
    {"episode": 2, "object": "red block", "target": "blue tray", "lighting": "dim", "seed": 12},
    {"episode": 3, "object": "red cube", "target": "blue tray", "lighting": "normal", "seed": 13},
])

print(card)
print(prompts)
print(panel)

Expected output: Prompting and conditioning embodied policies should leave a reproducible VLA evidence trace with checkpoint, action representation, robot interface, metric, and failure label.