Section 33.1: What LLMs can and cannot do in embodied tasks

"Ask an LLM what to do next and it answers fluently. Ask it whether the gripper is already closed and it guesses with equal fluency."

A Humbled Task Planner

Figure 33.1A previews the split this section develops in full: the LLM's strengths cluster around symbolic planning and language grounding, while its weaknesses cluster around precise spatial reasoning and high-frequency reactive control.

Read the figure as a boundary map. The LLM can propose task structure and language-level intent, but the embodied system still needs state estimation, affordance checks (where an affordance is a score for whether a proposed action is currently possible given the robot's pose, reach, and the object's position, not merely whether it sounds sensible), typed action APIs, execution monitoring, and a verifier that can reject unsupported plans.

Figure 33.1 makes that division concrete: it traces the closed loop from instruction to planner to tool API to verifier, then routes failure evidence back to the next decision. Every component discussed below maps onto one of its four stages, so the figure doubles as a structural template.

Tell a state-of-the-art LLM to "put the mug on the shelf" and it will hand you a flawless subtask list in under a second; tell it whether the gripper is currently open or closed and it will guess, with exactly the same confidence, and that asymmetry is the whole reason this section exists.

The practical question is not whether the LLM can describe the right action sequence, but whether that description can be turned into safe, low-latency, grounded behavior. A plan that reads correctly but cannot be executed is not a plan; it is a polite suggestion the robot has no way to honor.

Theory

A convenient decomposition is $$\pi(a_t \mid h_t) = \kappa\bigl(\phi_\text{LLM}(x_t, m_t), \hat s_t\bigr),$$ where \(\phi_\text{LLM}\) proposes a symbolic or programmatic plan from language context \(x_t\) and memory \(m_t\), while \(\kappa\) is the grounded executor that consumes the proposal together with the current state estimate \(\hat s_t\). The executor, not the LLM, owns physical validity.

This split explains both the promise and the limits. LLMs compress broad semantic priors into a small number of candidate subtasks or tool sequences. They do not directly measure friction, occlusion, latency, or actuator saturation; this is the language-physics gap, and it is why every LLM proposal needs a grounded verifier before execution. When papers claim strong embodied performance, the key question is how much the grounded stack contributes beyond the language model itself.

The gap matters physically because a robot cannot pause while the world waits. An arm mid-grasp that receives a semantically plausible but physically invalid next command has no safe default: it must halt, drop the object, or execute the command and risk collision. Latency compounds the problem. An LLM call takes tens to hundreds of milliseconds, far too slow for reactive feedback control at servo rates (typically 500 Hz to 1 kHz). At 1 kHz, the arm moves through roughly 20 control cycles while the LLM still generates its first token. The plan therefore arrives for a configuration the robot left 20 steps ago. Any plan that arrives late is a plan computed for a world that no longer exists.

Why token probability and feasibility diverge

The gap has a mechanical cause: LLMs train on text, so their tokens represent concepts, not physical states. The model learns that "the cup is full" co-occurs with "pour carefully," but it never sees joint torques, fluid dynamics, or camera depth. At inference the model samples plausible next tokens, not dynamically feasible trajectories. High token probability and physical feasibility are independent. That independence is why the verifier must be a separate grounded component.

Before reading further: if an LLM proposal scores in the top 1% of token probability, what is the probability it is physically executable on the robot in front of you? Take a guess, then keep reading.

Worked Example

Code Fragment 1 implements the smallest planner boundary: the LLM proposes one symbolic subgoal, but the executor only accepts it if the required tool and state preconditions are satisfied.

# Accept an LLM proposal only when the grounded stack can execute it.
# The executor checks state and tool availability before acting.
# This boundary keeps semantic planning separate from physical validity.
proposal = {"step": "pick(red_mug)", "required_tool": "grasp"}
state = {"target_visible": True, "toolbox": {"grasp", "place"}}

can_execute = state["target_visible"] and proposal["required_tool"] in state["toolbox"]
decision = "execute" if can_execute else "replan"

print({"proposal": proposal["step"], "can_execute": can_execute, "decision": decision})

The expected output is an executable proposal whose semantic content and grounded feasibility agree. If `proposal` looked sensible but `can_execute` were `False`, the correct diagnosis would be a grounding or interface failure rather than a language-understanding success.

Practical Recipe

The forced-tool-choice discipline above keeps a single proposal honest; the recipe that follows scales that same boundary into a repeatable build checklist for an entire planner.

1. Write down which variables the LLM sees and which variables only the grounded stack sees.
2. Require every LLM proposal to map into a typed action or code object.
3. Attach a verifier to every proposal so textual plausibility never counts as success by itself.
4. Measure latency separately for planning, execution, and recovery.
5. Benchmark against strong non-LLM baselines on the same task contract before claiming an embodied gain.

Since the research frontier above keeps moving the verifier inside the model, it becomes even harder to tell what the language component actually contributes, which makes the next discipline essential. A useful scientific discipline is to evaluate LLM contribution at the intervention boundary. Replace the LLM planner with a hand-written planner, a behavior tree, or a retrieval baseline while keeping execution fixed. If performance barely changes, the embodied value comes from the grounded stack, not from the language model.

This reframes the hype around end-to-end embodied agents. Emitting an action token is easy; the hard test is whether a model holds a physically valid internal state, meets timing budgets, and recovers from embodiment-specific failures. Most current systems still delegate those jobs to specialized modules.

Those tools each occupy one slot in the boundary, but a boundary is only trustworthy if you can audit what crossed it, which is exactly what the trace below records. Code Fragment 2 saves a planner trace with the fields needed for real evaluation. The key idea is that every proposal is logged alongside its verifier result and latency, so semantic fluency cannot hide execution failure. The trace below records a logprob value (the log-probability the model assigned its own output token, a proxy for text fluency, not physical feasibility) next to the verifier's decision, so the two can be compared directly.

1. Record the prompt context or task card that generated the proposal.
2. Store the typed action, the verifier result, and the state preconditions checked before execution.
3. Measure planning latency separately from skill-execution latency.
4. Tag failures as semantic, state-estimation, tool-interface, or controller failures.
5. Compare planner variants on the same execution stack and episode set.

# Log an LLM planner proposal as an auditable control trace.
# Every field needed to separate semantic quality from execution cost is recorded together.
import time

def log_planner_trace(prompt_context, planner_output, verifier_result, precheck_ms, exec_ms):
    return {
        "prompt_context": prompt_context,
        "planner_output": planner_output,
        "verifier": verifier_result,
        "latency_ms": {"plan": precheck_ms, "execute": exec_ms},
        "timestamp": time.time(),
    }

trace = log_planner_trace(
    prompt_context="pick up the sponge",
    planner_output={"step": "pick(sponge)", "logprob": -0.08},
    verifier_result="blocked",
    precheck_ms=210,
    exec_ms=0,
)
print(trace)

The expected output is a co-recorded control trace where the planner decision, the verifier evidence, and the latency measurement all point in the same direction. A useful negative case would keep the same structure but end with `result='blocked'` or `result='replan'`, which would let you localize the failure without replaying the whole episode.

If an embodied LLM system underperforms, first ask whether the LLM chose the wrong subgoal, whether the tool interface was incomplete, or whether the grounded stack could not realize an otherwise good plan. Those are very different scientific conclusions.