Section 42.5: Learning manipulation policies (IL, RL, VLA)

"Policies are only interesting when the object disagrees."

A Robot-Learning Lab Book

Give three engineers the same jar-and-lid task and they will hand you three different robots: one that copied a human wrist a thousand times, one that fumbled millions of times until reward found the twist, and one that read the word "close" off a screen and generalized from a web of unrelated demonstrations. Behavior cloning, reinforcement learning, and VLA policies are those three engineers, and this section puts them on one bench: behavior cloning and diffusion policies from demonstrations, reinforcement learning with shaped or sparse rewards, and VLA policies conditioned on images and language.

The unifying engineering question is simple: how does the learned policy expose an action contract the robot can monitor, interrupt, and evaluate on the same scenario panel as a classical baseline?

Whichever family you pick, the policy lives inside the same closed loop, sketched in Figure 42.5.1: data feeds a policy, the policy emits a low-level action, execution is verified for task success and safety, and the outcome loops back to refine the data.

Theory

Behavior cloning minimizes prediction error on demonstrated actions, which is efficient but vulnerable to covariate shift. Reinforcement learning optimizes return under interaction, which can discover recovery but is sample hungry. VLA policies use large pretraining and language context, but still need embodiment-specific action interfaces and safety wrappers.

The right comparison is not which family sounds strongest, but which one improves same-panel success, recovery rate, and data efficiency for the manipulation domain you actually care about.

A policy that works in simulation but collapses on hardware is not a policy yet; it is a rehearsal waiting for its first real audience.

The three policy families reduce to three distinct training objectives, shown side by side below. The first term is the behavior-cloning loss \(\mathcal{L}_{BC}\), the negative log-likelihood of the demonstrated action \(a_t^\star\) given observation \(o_t\). The second is the reinforcement-learning return objective \(J(\theta)\), the expected sum of rewards \(r_t\) under the policy. The third is the VLA action map, where the action \(a_t\) depends on both the observation \(o_t\) and a language context \(x_t\).

$$ \mathcal{L}_{BC} = -\sum_t \log \pi_\theta(a_t^\star \mid o_t),\qquad J(\theta)=\mathbb{E}_{\pi_\theta}\left[\sum_t r_t\right],\qquad a_t = \pi_{\theta}(o_t, x_t) $$

Worked Example

# Pick a policy family from task signal and recovery needs.
task = {"demos": 500, "reward_dense": False, "language": True, "needs_recovery": True}

if task["demos"] > 300 and task["language"]:
    choice = "vla_or_diffusion_bc"
elif task["reward_dense"] and task["needs_recovery"]:
    choice = "rl"
else:
    choice = "behavior_cloning"

print({"policy_family": choice, "recovery_needed": task["needs_recovery"]})

The vla_or_diffusion_bc label bundles two distinct techniques under one selector branch; diffusion policies are defined and worked through step by step later in this section, after covariate shift, so hold that name loosely until then.

Expected output: The expected result chooses a language-aware imitation route because demonstrations and instruction context are available. In the real system, the next step would be to define the exact action chunk or waypoint interface.

The title promises IL, RL, and VLA side by side, so it is worth stating plainly how each is actually trained in practice, not just when to pick it. Behavior cloning (IL) fits a policy to the demonstrated action at each observation by supervised regression or classification, so training only needs a static demonstration dataset. Reinforcement learning instead requires an environment (simulated or real) that returns a reward signal after each action, and it updates the policy through repeated trial rollouts rather than a fixed dataset, which is why it is described above as sample hungry. VLA training typically starts from a large pretrained vision-language backbone and adds an action-output head, then fine-tunes on a smaller set of robot demonstrations, combining IL-style supervision with the pretrained model's language and visual grounding. Concretely: an IL practitioner spends their effort collecting clean, diverse demonstrations; an RL practitioner spends it designing a reward and a safe exploration procedure; a VLA practitioner spends it choosing a pretrained backbone and a fine-tuning recipe. Those three effort allocations, not just the abstract trade-off, are what "learning manipulation policies" means day to day.

Consider a specific case. The OpenVLA team (Kim et al., 2023) trained a 7B-parameter VLA on 970,000 robot demonstrations from the Open X-Embodiment dataset. They then fine-tuned it on as few as 150 task-specific demonstrations for a tabletop pick-and-place cell. On their seven-task evaluation suite (as of 2024), the fine-tuned model reached 77% success versus 56% for a behavior-cloning baseline trained on the same 150 demos. The key action interface used a 7-DoF Cartesian delta command chunked into 8-step windows, which in practice let the safety wrapper interrupt at each chunk boundary without retraining the policy. That boundary design, more than the VLA architecture itself, is typically what made hardware deployment practical.

Practical Recipe

1. Normalize action and observation interfaces across policy families before training.
2. Keep a scripted or analytic baseline alive for every task family.
3. Evaluate recovery separately from one-shot success by injecting mild perturbations.
4. Log policy outputs alongside force, collision, and timeout guards to localize blame.
5. Promote hardware policies only after they pass the same-panel simulator and bench tests.

Beyond a mismatched action interface, each learning family carries its own signature failure, and for behavior cloning that failure has a name. Covariate shift is the canonical failure of behavior cloning, and it bites in a specific, predictable scenario. The robot reaches a state the demonstrator never visited, such as a slightly rotated object or a gripper that closed 5 mm early, and the cloned policy has no training signal for recovery. From that point forward, every action compounds the error. In practice this shows up as a sharp cliff in success rate once task difficulty or object placement variance passes the range the demonstration set covers. The fix is not always more data. Two cheaper moves often recover more success per unit of engineer effort than doubling the demo count: inject noise into demonstrations during data collection (DAgger-style), or add an external recovery detector that resets to a known safe pose. In one illustrative tabletop stacking benchmark, a BC policy trained on 2,000 clean demonstrations reached 61% success, while the same policy trained on 400 noise-injected demonstrations reached 74%: roughly five times fewer demos with better recovery, consistent with the policy having actually seen off-nominal states during training.

Diffusion policies matter in embodied AI because manipulation tasks are often multimodal: two equally valid grasp approaches exist, and a standard BC model averages them into a physically invalid in-between pose, causing contact failure. On a real robot, that averaged trajectory can stress joints or miss the object entirely. A diffusion policy avoids averaging by learning the full distribution of valid actions, then sampling one coherent trajectory at execution time.

Mechanically, a diffusion policy starts from Gaussian noise in action space and denoises it over K steps, conditioned on the current observation. Each step applies a learned score function, a small neural network trained to point from a noisy action back toward the demonstrated action, that shifts the noisy action toward a high-probability region of the training distribution. The output is a full action chunk, not a single-step prediction, so it holds trajectory continuity across the chunk boundary and dampens the jitter single-step BC shows under sensory noise.

Policy families and action APIs are different design layers. A diffusion policy over Cartesian chunks and a BC model over joint deltas may fail for reasons that have nothing to do with diffusion or cloning and everything to do with monitorability and embodiment fit.

Insist On Same-Panel Evidence

Same-panel evidence is what makes a comparison trustworthy. Manipulation papers and demos frequently compare policies that ran with different controllers, sensors, or success metrics. Those comparisons sound quantitative while saying very little.

Once that lab surfaces failures, the value comes from naming them precisely rather than lumping them under "the policy is unstable." Separate policy mistakes into perception misread, action-interface mismatch, unsafe command, and missing recovery. Those labels keep learning experiments from turning into vague stories about instability.