Section 33.3: Code as Policies: LLMs that write robot code

"When the planner writes code instead of text, the robot runs the plan instead of reading it. The compositionality is real; so is the gap between a correct sentence and a safe program."

A Policy Synthesizer With Commit Access

Read the figure as a generated-code safety boundary. Code as Policies is valuable only when generated functions are sandboxed, typed, checked against robot APIs, and traced from natural-language intent to executable motion calls.

Give a language model write access to a robot's action API and it will happily emit a five-line loop that stacks blocks into a pyramid; the unsettling part is that the same five lines can drive an arm into the table before any human reads them, so the real question is not whether code generation works for embodied control but how narrow, typed, and testable the runtime interface must be to make it survivable.

The practical question is what kind of code the model should be allowed to write and how that code should be checked before touching the robot.

Theory

Figure 33.3 names the four stages this theory has to account for: an instruction enters, a planner turns it into code, a tool API executes the calls, and a verifier feeds failure evidence back into the next decision. Keep that loop in mind as the formalism below makes each stage precise.

Instead of selecting one symbolic action, as an affordance-grounded planner does when it scores each next skill, the model emits a program \(P\) over a robot API \(\mathcal A\). The control loop becomes $$P = \phi_\text{LLM}(x, h_t), \qquad a_{t:t+H} = \operatorname{Exec}(P, \mathcal A, \hat s_t),$$ where safety now depends on the allowed API, runtime guards, and verification suite as much as on the model's semantic quality.

Why programs beat text plans

Code generation helps when tasks require loops, conditionals, and compositional reuse across subtasks. A free-text planner may say 'repeat until the drawer is closed'; a generated program can actually encode the loop condition. This is called the policy-as-program shift, and it changes what a planner can express. Liang et al. (2022) measured the gap on the same underlying model: on novel compositional instructions, code-based policies succeeded on roughly 90% of trials, while step-by-step text prompts succeeded on fewer than 50%. Counting the steps shows why the gap is so large. A flat text plan for a three-object sorting task must enumerate every conditional ("if the blue block is already placed, skip it") as a separate bullet, which typically produces 20 to 30 lines of instruction. The equivalent generated loop expresses the same logic in 5 to 8 lines, with no instructions duplicated across branches. The downside is that the model can also generate brittle logic or unsafe API sequences if the execution environment is too permissive.

This shift matters physically because a robot in the real world hits partial failures mid-task: a grasp slips, a detected object pose is stale, a conveyor is not yet clear. A flat list of actions cannot respond. It executes the next step regardless of outcome. A program can check a sensor return value, re-detect, and branch. Without that structure, a physical robot applies force to an already-placed object or repeats a motion that already failed, and the hardware and safety consequences are real.

The shift happens at the execution boundary. Given a typed API schema, the LLM writes a small function that calls those primitives directly instead of natural-language steps a separate parser must interpret. The runtime executes it, returns typed values, and the function's own control flow decides what happens next. The LLM writes once; the interpreter runs the result.

Those typed return values also close the loop back to sequential planning: a program's detect or pick call returns a status the LLM can branch on directly, so a code-as-policies controller can call into the same affordance-scored primitives that section 33.2's planner selects one at a time, just with the sequencing and error handling now expressed as code rather than re-planned by a separate module after every step.

Before reading the algorithm below, consider this: what exactly goes wrong when a generated program passes every whitelist check but still causes the robot to collide with the table? The answer shapes every design decision in the synthesis loop.

Worked Example

Code Fragment 1 generates a tiny skill program over a restricted API and then validates that every called function is allowed. It makes the interface boundary concrete rather than celebrating raw text generation.

# Validate that generated code calls only approved robot API functions.
# Program generation is useful only when the execution surface is constrained.
# A whitelist is the smallest possible runtime guard.
generated_calls = ["detect('red_mug')", "pick('red_mug')", "place('tray')"]
allowed = {"detect", "pick", "place", "wait"}

safe = all(call.split("(")[0] in allowed for call in generated_calls)
print({"calls": generated_calls, "safe": safe})

The expected output is a generated micro-program whose every call lies inside the approved API surface. The key fact is not only that `safe` is `True`, but that the plan has already been reduced to inspectable calls such as `detect`, `pick`, and `place`, which makes downstream verification possible.

Practical Recipe

1. Expose a narrow API that names only the skills and queries the robot is allowed to call. For a Franka Panda tabletop setup, this typically means six to ten primitives: detect(obj: str) -> Pose3D, pick(obj: str, grasp_width_m: float), place(target: str), push(obj: str, direction: Vec3, distance_m: float), and a small set of state queries. Anything beyond that surface area, including raw joint-angle commands or direct Cartesian velocity writes, should be inaccessible to the LLM entirely.
2. Generate code into a sandbox or domain-specific language (DSL, a small language restricted to the robot's own vocabulary of actions rather than general-purpose Python) rather than into unrestricted Python. On a real system such as Boston Dynamics Spot running ROS 2 (Robot Operating System 2, the standard middleware for passing commands and sensor data between robot software modules), a sandbox prevents the generated loop from issuing ros2 topic pub calls or writing to the joint-velocity controller directly, both of which bypass MoveIt 2 collision checking and, in practice, are the kind of unguarded call that has been reported to cause hardware damage in research deployments when a sandbox was not enforced.
3. Run static and runtime checks before sending any call to the robot. The static check should parse the generated AST for calls outside the whitelist; the runtime check should replay the call sequence in Isaac Sim or PyBullet, two physics simulators used to test robot code safely before it runs on hardware, against the current scene state before committing to hardware, adding roughly 200 to 400 ms of latency that, in practice, catches most pick failures caused by stale object poses before they reach hardware (the replay only helps to the extent the simulated scene state matches the real one).
4. Log the generated program and the verifier result together. A task-success bit alone cannot distinguish a Pydantic argument-order error from a genuine grasping failure; the full program text and the verifier trace make repair deterministic rather than diagnostic.
5. Treat repair and regeneration as first-class parts of the loop rather than as exceptional events. In the original Code as Policies experiments (Liang et al., 2022), roughly 30 to 40 percent of first-pass generations required at least one repair cycle before passing the whitelist check; systems that did not budget for this latency exceeded their real-time planning windows on tasks with more than three subtasks.

Code generation changes the abstraction level of planning. Instead of choosing the next action only, the model can synthesize local control flow and data flow. That is why program-based interfaces often generalize better than step-wise prompts on long tasks with repeated patterns.

The cost is that verification must move closer to software engineering. You need typed signatures, unit tests, API whitelists, and runtime contracts, not just high-level task metrics. A generated program is a real artifact, and it deserves real software scrutiny before it reaches hardware.

Choosing those tools is only half the discipline; the other half is recording what each generated program did so the choices can be evaluated. Code Fragment 2 stores the generated program and its verifier result in one record. That is the right unit for ablations because it lets you compare program quality, execution success, and repair frequency together.

1. Save the generated program text or AST in the experiment artifact.
2. Run signature checks, whitelist checks, and simple execution tests before deployment.
3. Keep the generated program short enough that a human can audit it during development.
4. If verification fails, route the error message back into a regeneration step rather than guessing a patch silently.
5. Compare program-generation systems on the same API surface and same robot backend.

# Store the generated program and verifier result as one experiment record.
# Keeping only a task-success bit loses the information needed to diagnose
# argument errors, loop-termination bugs, and API misuse.
import json, pathlib, datetime

def save_execution_record(
    instruction: str,
    program_text: str,
    api_schema: dict,
    verifier_errors: list,
    execution_trace: list,
    task_success: bool,
    output_dir: str = "experiment_records",
) -> pathlib.Path:
    record = {
        "timestamp": datetime.datetime.utcnow().isoformat(),
        "instruction": instruction,
        "program_text": program_text,
        "api_schema_keys": list(api_schema.keys()),
        "verifier_errors": verifier_errors,
        "execution_trace": execution_trace,
        "task_success": task_success,
    }
    out = pathlib.Path(output_dir)
    out.mkdir(parents=True, exist_ok=True)
    path = out / f"record_{record['timestamp'].replace(':', '-')}.json"
    path.write_text(json.dumps(record, indent=2))
    return path

# Example call
path = save_execution_record(
    instruction="Sort red blocks by size and stack left to right.",
    program_text="for obj in detect_all('red_block'): pick(obj); place('stack')",
    api_schema={"detect_all": "str -> list[str]", "pick": "str -> None", "place": "str -> None"},
    verifier_errors=[],
    execution_trace=["pick(block_s)", "place('stack')", "pick(block_m)", "place('stack')"],
    task_success=True,
)
print(f"Record saved to {path}")

The expected output is an execution record that ties the literal generated program to the verifier outcome and the observed execution result. That linkage matters because code-generating agents often fail through argument misuse or illegal call order, and those errors are invisible if you keep only a task-success bit.

Once that record exists, it becomes the lens for reading failures, because it lets you sort what went wrong into two very different bins. When code-based planners fail, separate semantic plan errors from software-interface errors. The model may understand the task but still misuse an argument order, forget a termination condition, or violate a runtime precondition.

A program that passes every whitelist check but has no termination guarantee is not a policy: it is a loop waiting for a deadline to miss.