Section 26.4: Language as a high-level controller

"Language is the only high-level controller that can be corrected with more language."

Section 26.4

Why Hierarchy Matters

A balance controller that misses its deadline by ten milliseconds drops a quadruped on its face, yet a language planner that takes a full second to pick the next skill is perfectly fine: how can one controller serve both clocks at once? It cannot. That mismatch is why hierarchy separates timing, contact, recovery, and sequencing, so the planner can select skills without pretending every low-level policy is deterministic. On a Franka Panda arm, a grasp policy runs its impedance controller at 1 kHz on the joint torque interface. The SayCan-style language planner (SayCan is the system, detailed later in this section, that scores candidate skills by combining a language model's judgment with a learned affordance check) above it fires once per skill at roughly 1 Hz. Collapsing those two rates into one flat policy is what made pre-2020 systems brittle. The same split lets a Boston Dynamics Spot quadruped run a 500 Hz balance controller underneath a navigation skill that the planner treats as a single named action, so the high-level layer never has to reason about foot placement to dispatch "go to the kitchen."

Figure 26.4A shows the overall shape of this idea: an LLM turns one instruction into a sequence of named skills drawn from a library, and each skill in turn runs a low-level policy that closes the physical loop. Language can propose task decompositions, but it must be grounded into a typed skill library. The sentence 'inspect the shelf and bring the red cup' becomes executable only after perception resolves objects, affordance checks confirm feasible skills, and the planner maps words to verified actions. Without affordance grounding, an LLM planner selects an infeasible skill on roughly 40% of real kitchen trials; with grounding, that figure drops below 5% (SayCan, 2022). This is what makes affordance-gated language planning the architectural turning point between a chatbot that talks about tasks and a robot that completes them.

Formal Contract

The option tuple becomes a real audit checklist when each field holds a physical quantity. Consider a Franka Panda arm receiving the command "hand me the marker." The initiation set requires three sensor predicates: wrist force below 2 N, depth camera confidence above 0.85 on the target centroid, and base localization error below 3 cm. A visual-servoing impedance controller runs the internal policy at 1 kHz on the joint torque interface. Termination fires when the gripper force sensor reads above 8 N for 0.3 s, indicating a stable grasp, or when a 5-second timeout expires. The verifier then reads the wrist-mounted tactile array. If contact pressure spreads across at least 3 taxels (individual pressure-sensing elements in the tactile array, the tactile equivalent of camera pixels), the postcondition "object grasped" is confirmed in sensor space and the planner advances to the next skill.

The scoring rule below makes that audit checklist concrete: it assigns each candidate skill a probability that combines a learned language-grounding score with a hard affordance gate. Figure 26.4B, which follows the scoring rule and its worked example, traces this same flow end to end: a mission goal expands into a task graph, the task graph dispatches concrete skills, and a verifier gates each one before the plan is trusted.

$$p(\omega\mid u,o) \propto \exp(f_\theta(u,o,\omega))\,\mathbf{1}[\mathrm{affordance}(o,\omega)=1].$$

In this scoring rule, \(u\) is the tokenized language instruction, \(o\) is the current observation (RGB-D frame plus joint state), and \(\omega\) ranges over the named skills in the typed library. The affordance indicator \(\mathbf{1}[\mathrm{affordance}(o,\omega)=1]\) hard-blocks skills whose preconditions fail in sensor space: a grasp_marker skill is zeroed out if the depth point cloud returns no cluster within the arm's 855 mm reach envelope, regardless of how confidently the LLM scores it. On a Hello Robot Stretch platform operating in a real kitchen, this gating step has been observed (informal internal testing, not a controlled published benchmark) to eliminate roughly 35% of LLM-top-ranked candidates before any motion is attempted, preventing wasted cycles and collision-prone arm extensions.

Worked Implementation

That same gate, which on paper zeroes out infeasible skills, becomes a handful of concrete predicate checks once you write it in code, so the worked example below makes the affordance filter executable rather than conceptual.

Code Fragment 1 for Language as a high-level controller should expose initiation, progress, termination, verification, and failure reporting before connecting the skill to ROS 2, BehaviorTree.CPP, Drake, or a learned policy.

# Ground a language request into a typed skill sequence.
# The affordance check blocks skills that are not feasible in the current scene.
skills = {
    "inspect": {"requires": "camera_ready"},
    "navigate": {"requires": "map_ready"},
    "grasp": {"requires": "object_reachable"},
}
scene = {"camera_ready": True, "map_ready": True, "object_reachable": False}
request = ["inspect", "navigate", "grasp"]
plan = []
for skill in request:
    requirement = skills[skill]["requires"]
    plan.append((skill, "allowed" if scene[requirement] else "blocked"))
print(plan)

This expected output should be read as a grounded execution filter for language, not a language model answer by itself. The important outcome is that grasp stays blocked until the world state changes, which prevents fluent language from bypassing reachability constraints.

Why a Typed Skill Library

A typed skill library matters because physical robots have heterogeneous action spaces: a gripper skill consumes a 6-DOF pose target (a target position and orientation in three-dimensional space: three coordinates for location plus three for rotation, hence "6 degrees of freedom"), a navigation skill consumes a 2D goal on a costmap (a grid map where each cell's value reflects how costly or risky it is to drive through, used to plan routes around obstacles), and a force-control skill consumes a Cartesian wrench vector (a force-and-torque command expressed in x, y, z axes, distinct from the joint-angle commands a gripper skill expects). Without explicit types, a language planner can chain two skills whose output and input formats are incompatible, producing a silent failure at the physical interface rather than a detectable planning error. Catching a type mismatch at plan-construction time costs one predicate check; catching it after an arm has moved costs a recovery cycle and risks contact damage.

The mechanism annotates each skill entry with a signature: input type (e.g., ObjectPose), precondition predicates (e.g., reachable(obj)), and output type (e.g., GraspedObject). The planner treats skill chaining as type unification. The output type of skill \(k\) must match the input type of skill \(k+1\), and the current world state must satisfy every precondition predicate. This turns language planning from free-form text generation into a type-checked composition, so the planner rejects infeasible sequences before issuing any motor command. In practice, a type check costs one predicate evaluation taking under a millisecond; the recovery cycle it prevents costs 8 to 12 seconds of arm retraction, re-localization, and re-approach, so the type system pays for itself after a single avoided failure.

Practical Recipe

Typing, affordance gating, and the verifier contract only pay off when they are baked into a skill from the start, so the following recipe orders those design decisions into the sequence you would actually follow when building a new skill library.

1. Name each skill with a verb and object: navigate_to_station, grasp_handle, dock_drone, or change_lane.
2. Write preconditions, postconditions, safety guards, timeout, and recovery behavior before training a policy.
3. Represent sequencing as a finite-state graph, behavior tree, or task-and-motion plan so failures have explicit routes.
4. Use language as a planner only after commands are grounded into a typed skill library with affordance checks.
5. Evaluate composition, not only individual success. Many failures occur when two correct skills meet at a bad boundary.