Section 3.7: Where LLMs, VLMs, and VLAs sit in the stack

"A foundation model is a powerful component, not a complete robot. The stack still needs every layer the model does not replace."

A Careful Control Loop

Pick the wrong foundation model for a layer and you can spend 50,000 demonstrations training a robot to do what 300 would have taught the right architecture: the same "see the object, then move the gripper" gap costs almost nothing when one model owns it and balloons when a silent handoff sits in the middle. By the end of this section you should be able to assign any foundation model to the correct interface slot in a control stack and justify that choice from input modality, output contract, and latency budget. First we define the object of study, then we connect it to the agent loop (the perception-decision-action cycle introduced in section 3.1, in which each new observation depends on the last action taken), then we test it with a compact implementation. Figure 3.7 captures that cycle as the evidence, decision, consequence pattern that the three model families plug into at different points.

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Once the interface is inspectable, the next question is which model family belongs at each interface, and the answer is rarely the most advanced one. LLMs, VLMs, and VLAs are not interchangeable upgrades. They sit at different interfaces in the stack, as Figure 3.7A illustrates. An LLM maps language context to language, plans, tool calls, or symbolic instructions. A VLM maps images and language to grounded descriptions, detections, affordances, or decisions. A VLA maps visual and language context closer to robot action, often as action tokens, trajectories, or low-level command chunks. The gap matters in practice: a concrete named system, RT-2 (introduced fully below in the "Named Systems in Practice" callout, a Google robotics model that fuses a vision-language backbone with action-token output), reports roughly 3x the success rate of a language-only planner on novel object tasks in its original evaluation. The VLA earns that lead not because its underlying language model is smarter, but because it closes the interface between scene pixels and motor commands in a single forward pass. A language-only planner instead routes through a separate grounding step that can silently fail.

Action tokens are discrete vocabulary entries that a VLA predicts the same way a language model predicts the next word. Each token encodes a quantized motor command: a joint angle delta, a gripper width, or a Cartesian displacement. This matters in embodied AI because physical actuators have hard timing deadlines. A Franka Panda impedance controller (a low-level loop that regulates the relationship between joint force and position rather than commanding position alone) requires commands at 1 kHz, that is, one command every millisecond, so any model that cannot tokenize and decode within that window cannot close the loop safely. Representing actions as tokens lets a VLA reuse the transformer's generation machinery and keep grounding and execution in one differentiable pass. That single pass eliminates a silent-failure boundary where a symbolic plan could request an object the controller has no representation for. The cost of that boundary is not abstract, though the exact multiplier is illustrative rather than a fixed law: in reported case studies, bridging the same "point at object, then command controller" gap through a separate grounding step has typically required on the order of tens of thousands of demonstration episodes to train reliably, while a co-trained VLA that collapses both steps has reached comparable reliability with roughly two orders of magnitude fewer, because the gradient flows from motor error all the way back through the visual encoder without passing through a handoff that stops learning.

The design question is therefore not "Which model is most capable?" The better question is which interface needs learned generalization, and which interface still needs an explicit contract. A strong architecture often uses an LLM for task decomposition, a VLM for scene grounding, and a VLA or controller for execution, with explicit checks between them, an arrangement that mirrors the layered control hierarchy from deliberation down to reactive loops.

Worked Example

Before reading the code below, make a prediction: if the VLM fails to localize the target object, does the error surface in the LLM's plan, in the VLA's action output, or silently as a physically plausible but wrong motion?

The three model families compose into one pipeline: an LLM decomposes the goal, a VLM grounds each subgoal in the image, and a VLA turns a grounded subgoal into an action. The example uses stand-in functions for each model so the structure is visible, then runs the section's key diagnostic, oracle substitution, to locate which interface caused a failure. This is the executable form of the Fun Note: the plan and the pointing only matter if the action survives contact.

# Stand-ins for the three model families. Swap in real models later;
# the contract (what enters, what leaves) is what we are testing.
def llm_plan(goal):                       # language -> ordered subgoals
    return {"put apple in bowl": ["grasp apple", "place in bowl"]}[goal]

def vlm_ground(subgoal, scene):           # image+text -> object + pose
    obj = subgoal.split()[-1]             # "grasp apple" / "place in bowl"
    return scene.get(obj)                 # None if not visible -> grounding gap

def vla_act(grounded):                    # state+goal -> action token
    if grounded is None:
        return None
    return f"move_to({grounded['x']:.2f},{grounded['y']:.2f}) + close"

def run(goal, scene):
    for sub in llm_plan(goal):
        g = vlm_ground(sub, scene)
        a = vla_act(g)
        print(f"  {sub:16s} ground={g} action={a}")
        if a is None:
            return f"FAIL at grounding: {sub}"
    return "ok"

scene_ok  = {"apple": {"x": 0.4, "y": 0.1}, "bowl": {"x": 0.7, "y": 0.0}}
scene_bad = {"bowl": {"x": 0.7, "y": 0.0}}            # apple occluded

print("scene_ok :", run("put apple in bowl", scene_ok))
print("scene_bad:", run("put apple in bowl", scene_bad))
# Oracle substitution: inject the apple pose to confirm the VLM, not the
# LLM or VLA, was the first cause in scene_bad.
scene_fixed = dict(scene_bad, apple={"x": 0.4, "y": 0.1})
print("oracle   :", run("put apple in bowl", scene_fixed))

Expected output: the good scene runs end to end; the bad scene fails at grounding because the apple is not visible, and the VLA correctly produces no action rather than reaching for nothing; the oracle run succeeds once the missing pose is injected, which proves the VLM grounding was the first cause. The lesson matches the section's diagnostic: replace one model output with a verified value and see whether the system recovers, instead of swapping the whole foundation model on a hunch.

Practical Recipe

The oracle test that closed the worked example only pays off if the surrounding pipeline is built to expose interfaces in the first place, so the recipe below turns that diagnostic discipline into a build order.

1. Write the sensor-to-actuator contract before choosing a model: name the input modality (e.g., 640x480 RGB at 30 Hz from a wrist-mounted RealSense D435), the action representation (e.g., 7-DoF delta joint angles at 50 Hz for a Franka Panda), and the success criterion in physical units (object within 5 mm of target pose, verified by force/torque sensor reading below 5 N).
2. Build the simplest possible closed-loop baseline first. For pick-and-place, a single VLM call that returns a 2-D pixel centroid plus a fixed-height grasp heuristic will expose grounding failures before a VLA is ever loaded.
3. Run the pipeline in simulation (MuJoCo or Isaac Sim) with a fixed random seed to get a reproducible failure rate before touching real hardware. Sim-to-real transfer errors compound; find them cheaply.
4. Record failures with physical attribution: perception gap (object not detected), pose error (localized but wrong 6-DoF), timing violation (plan latency exceeded controller deadline, typically 8-20 ms for impedance control), actuator saturation (commanded torque exceeded joint limit), or recovery failure (fallback held position but operator was not notified).
5. Run at least one structured perturbation: change the lighting by 500 lux, add a distractor object in the same color category as the target, or introduce a 50 ms latency spike on the grounding interface. Each perturbation probes a different layer of the LLM/VLM/VLA stack.

Where LLMs, VLMs, and VLAs sit in the stack becomes useful when it is tied to a closed-loop contract for how perception, estimation, planning, learning, and control are arranged into a system. The contract names the observation stream, the action representation, the timing budget, the safety boundary, and the result artifact. That is the bridge between a readable concept and a system a skeptical builder can test.

Keep three claims separate: the conceptual claim, the systems claim, and the evidence claim. A good explanation, a clean API, and one successful rollout each prove something different, and conflating them hides where an architecture actually breaks.

A robust implementation starts with one inspectable baseline whose artifact records observations, actions, units, timestamps, seeds, termination reasons, and the perturbation applied. The maintained-tool version is useful only if it preserves that schema and lets the comparison remain construct-matched.

1. Write a one-paragraph task contract with observation, action, success, failure, and safety fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save one artifact containing configuration, seed, metrics, traces, and failure labels.
5. Compare methods only when the same script evaluates the same panel, split, seed set, and metric.

When Where LLMs, VLMs, and VLAs sit in the stack fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

Diagnose the interface, not the model

The quickest diagnostic is to replace one model output with a verified oracle value. If the system succeeds when the LLM plan is manually corrected, the language-level decomposition is the first suspect. If it succeeds only when the VLM grounding is corrected, the problem is scene understanding. If it still fails after both are corrected, inspect action scaling, controller limits, and embodiment-specific assumptions before changing the foundation model.

Decision heuristic, in one pass: for any new architecture, ask three questions in order. (1) Does this layer need open-ended language generalization (novel goals, novel phrasing)? If yes, that layer wants an LLM. (2) Does this layer need to turn pixels into grounded scene state (identity, pose, affordance)? If yes, that layer wants a VLM. (3) Does this layer need to emit motor commands within a hard timing deadline? If yes, that layer wants a VLA or a classical controller, not a general-purpose LLM or VLM. Apply the interface-risk column of the comparison table above to whichever family the answer selects, and the assignment is complete.