Section 33.2: SayCan: affordance-grounded planning

"The language model knows what sounds like a good idea. The value function knows what the room will actually permit. SayCan is the negotiation between them."

Section 33.2

Read the figure as the SayCan product rule in system form: language likelihood proposes what is useful, affordance likelihood estimates what is possible, and the robot acts only where both scores support an executable skill.

Ask a fluent language model what a thirsty person wants and it will confidently say "pick up the soda can," even when that can sits on a shelf the robot's arm cannot reach and an open water bottle waits at its fingertips: the words are right and the physics is wrong. An affordance names what a skill can actually accomplish in the current physical state: whether the object is reachable, graspable, and the action executable right now, as opposed to merely sounding correct. SayCan is the natural antidote to this gap, because it makes the combination of semantic relevance and executability stronger than either source of evidence alone. Figure 33.2 traces the closed loop this grounding sits inside, where an instruction reaches a planner, the planner commits to a skill through the tool API, and a verifier feeds failure evidence back into the next decision.

A planner that speaks fluently but cannot reach the shelf is not a robot controller; it is a suggestion box.

The practical question is how to combine semantic relevance and executability without letting one wash out the other.

Theory

SayCan scores each candidate skill \(k\) with a language prior and an affordance value: $$k^* = \arg\max_k \; p_\text{LLM}(k \mid x, h_t) \cdot V_k(s_t).$$ The language term prefers semantically appropriate next steps, while the value function estimates whether the robot can execute that step successfully in the current state.

The multiplication matters. A skill with high semantic probability but near-zero affordance should be rejected, and a highly executable skill with no semantic relevance should not dominate just because it is easy. This is called the language-times-affordance product rule, and in the original SayCan experiments on real kitchen hardware it typically lifted task completion from roughly 56% (language score alone) to 74% (joint score), a gap reported across 101 real-robot trials in that study. The method therefore depends on score calibration and on the quality of the candidate skill library.

Why the affordance value carries physical constraints

The value function \(V_k(s_t)\) matters for embodied AI because robots face hard physical constraints that a text-trained language model cannot observe. Joint limits, reach envelopes, friction coefficients, and object occlusion all determine whether a skill is physically executable in a given configuration. A wrong choice carries real consequences: dropped objects, collision, or wasted time that the physical world will not roll back. A scalar learned on real interaction data keeps those constraints implicit yet grounds them in actual robot experience rather than language statistics.

Mechanically, each \(V_k\) is a per-skill success-value function trained with reinforcement learning (RL) or behavioral cloning on robot episodes. Given the current sensor observation \(s_t\), it estimates the probability of completing skill \(k\) from that state. Training on real or simulated executions lets it capture preconditions such as object reachability, grasp clearance, and surface stability, and it does so without any explicit symbolic encoding of those constraints.

The data asymmetry sharpens the point. The original SayCan system needed roughly 68,000 robot episodes across 551 skills to build the affordance library, while the language model required zero additional robot interaction and contributed its entire semantic prior from pre-training on text alone.

Worked Example

The ten-step algorithm above is easier to trust once you watch its scoring core run on concrete numbers, so the next fragment strips the loop down to the single product computation at its heart.

Code Fragment 1 implements the core SayCan score on three skills. The example is tiny, but it makes the product structure visible and shows how the selected action can differ from the highest language score alone.

# Combine semantic plausibility with grounded affordance values.
# The best skill is not the one with the largest language score alone.
# Product scoring removes semantically attractive but infeasible actions.
skills = {
    "pick_sponge": {"p_llm": 0.55, "affordance": 0.92},
    "turn_on_sink": {"p_llm": 0.30, "affordance": 0.95},
    "wipe_spill": {"p_llm": 0.80, "affordance": 0.18},
}

combined = {name: round(v["p_llm"] * v["affordance"], 3) for name, v in skills.items()}
print(combined)
print(max(combined, key=combined.get))

The expected output is a ranking where the chosen skill is not merely the most semantically plausible sentence completion, but the action with the highest joint semantic and affordance score. Here `pick_sponge` wins because it is both relevant and executable in the present scene, while `wipe_spill` is semantically tempting but prematurely chosen.

Practical Recipe

1. Define a compact skill library whose actions expose clear preconditions and effects.
2. Generate only semantically plausible skill candidates rather than scoring the entire API surface.
3. Estimate affordance or value in the current state before execution, not from a stale scene snapshot.
4. Normalize or calibrate the two scores so one term does not dominate by scale alone.
5. Inspect failure cases where the right long-horizon plan starts with a low-probability semantic step.

Knowing where a losing skill belongs in the trace only helps if its two scores are comparable, which exposes the deeper issue: calibration. The product formula is meaningful only when the two terms share an interpretation. A miscalibrated value model swamps the semantic term and reduces the planner to greedily grabbing whichever skill is easiest right now.

The method also inherits the classic option-discovery problem from hierarchical RL. It can only select among skills it already knows. If the correct subtask is missing from the library, no amount of language fluency will recover it, which is why skill design and language as a high-level controller remain central.

Whichever combination of these tools you assemble, the planner is only debuggable if it records why it chose what it chose, which is what the next fragment captures. Code Fragment 2 stores the separate scores as an audit artifact rather than only the winning skill. This is the minimum needed to understand whether the semantic prior or the affordance estimator caused a bad decision. The steps below map directly onto the code that follows:

1. Log the candidate skills and both scores for each decision point.
2. Keep the value-estimation state snapshot or seed so scores can be reproduced.
3. Store the chosen skill and the first rejected alternative for debugging.
4. Measure how often the affordance term changes the top language choice.
5. Benchmark with the same skill library and same execution stack when comparing alternatives.

# Build one audit record per decision point instead of logging only the winner.
# Keeping both raw scores lets a failure be traced to language or affordance.
def audit_record(candidates, p_llm, affordance):
    scored = {
        name: round(p_llm[name] * affordance[name], 3) for name in candidates
    }
    ranked = sorted(scored, key=scored.get, reverse=True)
    chosen, runner_up = ranked[0], ranked[1]
    return {
        "chosen": chosen,
        "chosen_score": scored[chosen],
        "runner_up": runner_up,
        "runner_up_score": scored[runner_up],
        "p_llm": p_llm[chosen],
        "affordance": affordance[chosen],
    }

candidates = ["pick_sponge", "turn_on_sink", "wipe_spill"]
p_llm = {"pick_sponge": 0.55, "turn_on_sink": 0.30, "wipe_spill": 0.80}
affordance = {"pick_sponge": 0.92, "turn_on_sink": 0.95, "wipe_spill": 0.18}
print(audit_record(candidates, p_llm, affordance))

The expected output is an audit tuple that preserves both factors of the SayCan product. If a future run selected the wrong skill with a high semantic score but a weak affordance score, the repair target would be calibration or candidate generation rather than the planner prompt alone.

If the chosen skill is poor, first check candidate generation, then affordance calibration, then library coverage. SayCan errors often come from what is missing from the candidate set, not only from how the final score is computed.