Section 3.3: End-to-end learned policy pipeline

"The end-to-end policy learned something. What it learned is between the gradients and the weights, and neither will tell you."

A Diagnostic Engineer With No Intermediate Representations

Strip out the depth estimator, the grasp planner, and every "if-then" rule a roboticist would normally write, replace the whole stack with one neural network from pixels to torques, and you get a policy that grasps mugs it has never seen, yet offers nothing to inspect the moment it fails. That is the bargain of the end-to-end learned policy pipeline, and this section makes it usable. Figure 3.3A shows the shape of the idea: raw camera pixels enter a single neural network that directly outputs motor torques, with learned layers replacing every explicit intermediate module. Figure 3.3 reframes the same architecture as a closed-loop evidence, decision, consequence pattern, where each action changes the next observation. The section defines the object of study, connects it to the agent loop, and tests it with a compact implementation.

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

The word "pipeline" in this section's title does not mean a chain of separately-coded modules; it means the sequence of stages a builder still has to run even when a single network makes the moment-to-moment decision: collect demonstrations, normalize actions, train the network, and audit coverage before deployment. Each of those stages appears later in this section (the Algorithm callout below walks through exactly these steps), so "end-to-end" describes the runtime decision, while "pipeline" describes the surrounding workflow that produces and checks that decision.

Theory

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

An end-to-end policy is motivated by the handoff problem in modular stacks: if perception produces exactly the wrong abstraction, the planner never sees the information it needed. Instead of committing to intermediate symbols, the policy learns a direct map from observations and goals to actions:

$$a_t = \pi_\theta(o_{\le t}, g), \qquad \theta^\star = \arg\min_\theta \sum_{(o,g,a^\star)} \ell(\pi_\theta(o,g), a^\star).$$

The loss \(\ell\) is often a regression loss for continuous actions, a cross-entropy loss for discrete action tokens, or a diffusion-style denoising loss for action chunks. The benefit is representation freedom: the model can keep visual, temporal, and language cues that a hand-written state estimator might discard. The cost is diagnostic opacity without intermediate symbols. A policy that cannot be inspected module by module is typically not less fragile; it is fragile in a way that is harder to see. When the policy fails, the builder must probe data coverage, action scaling, temporal context, embodiment mismatch, and distribution shift rather than open a single broken module. The coverage stakes are concrete. Behavior-cloning evaluations through 2024 (ACT, Zhao et al. 2023; OpenVLA, Kim et al. 2024) show that a policy trained on roughly 50 in-distribution demonstrations reaches 80-90% task success within that distribution. The same policy drops below 20% on observations roughly 2x outside the training range. In these reported evaluations, that gap tracks data coverage far more closely than model capacity (though the two are not fully separable, since a larger model can also partially compensate for sparse coverage). Adding around 150 diverse demonstrations recovers performance above 70% with no architecture change in those same evaluations. In other words, in this evidence, the difference between a robot that works and one that fails is closer to 150 extra teleoperated episodes than to a bigger network.

The algorithm above is a template; real published systems fill in its steps with different network sizes and datasets, which is why the numbers below vary so much between systems. Consider a specific case. RT-2 (Brohan et al., 2023) fine-tunes a 55-billion-parameter vision-language model to output robot joint commands as text tokens. By transferring web-scale visual semantics, it reaches 62% success on novel objects and categories absent from the robot training data. Action Chunking with Transformers (ACT, Zhao et al., 2023) takes a different point on the design curve. A smaller 80-million-parameter transformer trains on 50 human demonstrations per task and hits 80-90% success on precise bimanual manipulation. It predicts 100-millisecond action chunks (short bursts of future actions produced in one forward pass, defined in detail two paragraphs below) that smooth out compounding errors. OpenVLA (Kim et al., 2024) shows the same vision-language-action (VLA) recipe scales to open weights at 7 billion parameters, matching RT-2 manipulation success at roughly one-tenth the inference cost. These three systems span the specificity-versus-cost tradeoff the formula above leaves open. The observation space and action tokenization differ, but the direct \(o \to a\) mapping stays constant across all three.

ACT earned its strong numbers above by predicting short action chunks rather than single steps. That single design choice accounts for much of the reported gain. The ACT paper's own ablations attribute the largest single improvement to chunking, though the final numbers also depend on the transformer architecture and the training data, so "most of the work" should be read as an ablation-supported estimate, not an exact decomposition. Action chunking matters in embodied AI because physical robots accumulate error over time: each single-step prediction adds noise, and the next prediction is conditioned on the now-corrupted state. On a real manipulator, this compounding causes the gripper to drift off the grasp axis within seconds. Predicting a short sequence of future actions in one forward pass breaks that feedback loop for the duration of the chunk, keeping the robot on a locally consistent trajectory without querying the policy at every timestep.

The mechanism works by changing the output head from a single action vector to a sequence of \(H\) consecutive actions. During training, the policy minimizes loss across all \(H\) steps jointly, so the network learns temporally consistent motion. At inference, the robot executes the chunk open-loop for \(H\) steps, then re-queries the policy. ACT uses a transformer encoder-decoder where the decoder autoregressively generates the chunk conditioned on a visual and proprioceptive token, where proprioception is the robot's sense of its own joint angles and gripper state, then executes all \(H\) outputs before the next observation is needed.

Worked Example

An end-to-end policy learns the map \(a = \pi_\theta(o, g)\) directly from demonstrations, with no hand-written state in between. The example fits a minimal behavior-cloning policy by least squares, then makes the section's central point: when it fails, the first diagnostic is not "open a module" but "audit data coverage." We do that with a nearest-neighbor check against the training set.

import numpy as np
rng = np.random.default_rng(0)

# Demonstrations: observation = [obj_x, obj_y], goal g = 1 (pick).
# Expert action = unit vector from a fixed gripper origin to the object.
N = 200
O = rng.uniform(-1, 1, size=(N, 2))          # objects in a seen region
A = O / np.linalg.norm(O, axis=1, keepdims=True)   # expert reach direction

# Behavior cloning by least squares: a = O @ W  (the whole "training run").
W, *_ = np.linalg.lstsq(O, A, rcond=None)

def policy(o):
    return o @ W

def coverage(o, k=5):                          # distance to k nearest demos
    d = np.linalg.norm(O - o, axis=1)
    return np.sort(d)[:k].mean()

for label, o in [("in-distribution", np.array([0.3, -0.4])),
                 ("far OOD",        np.array([3.0,  3.0]))]:
    a = policy(o)
    err = np.linalg.norm(a - o / np.linalg.norm(o))
    print(f"{label:16s} cover={coverage(o):.2f} action_err={err:.2f}")

Expected output: the in-distribution query has small coverage distance and bounded action error; the far out-of-distribution query has a large coverage distance and a much larger error (the linear policy cannot extrapolate the normalized reach direction it never saw). That contrast is the diagnosis: the policy is not "broken," it is being asked about a region it never saw. This is why the first isolation test for an end-to-end policy is a coverage audit, not a module trace, because the architecture deliberately removed the modules you would otherwise inspect.

Practical Recipe

1. Fix the observation contract before touching model architecture: specify camera resolution (e.g., 224x224 RGB at 10 Hz for ACT, 480x640 for RT-2), whether proprioception (joint angles, gripper state) is concatenated to the visual token, and the maximum acceptable sensor latency. A 20 ms timestamp mismatch between a wrist camera and joint encoders will corrupt the action-chunking temporal alignment and produce oscillating Franka Panda wrist commands even when the visual prediction is correct.
2. Use a behavior-cloning baseline trained on 50 demonstrations as the first checkpoint, not a pretrained VLA. On a Franka tabletop pick-and-place task this baseline achieves 70-80% success and is fast enough to retrain overnight, making data-coverage failures visible before investing in fine-tuning a 7B-parameter OpenVLA checkpoint.
3. Before adding more demonstrations, ask: what did the policy never see? A coverage audit on the failed rollouts typically reveals that the missing cases cluster in one corner of the workspace, one lighting condition, or one object orientation. Fixing that gap costs far less than doubling the total dataset size.
4. Normalize every action dimension to zero mean and unit variance using per-joint statistics computed from your demonstration dataset, not from a generic prior. Store the normalization constants in the checkpoint. Franka joints span roughly -2.9 to 2.9 rad but finger joints span only 0 to 0.08 m; without per-joint normalization the output head saturates on shoulder joints and undercontrols the gripper, causing drops on pick tasks even when reach accuracy is good.
5. Record failures with four fields: (a) nearest-neighbor coverage distance to the training set in observation space, (b) which joint or gripper axis shows the largest tracking error, (c) whether the episode terminated from a contact event or a timeout, and (d) whether the scene lighting or object pose was inside the distribution of training images. These four fields separate coverage failures, control failures, and sim-to-real appearance failures without opening the model weights.
6. Before deploying on physical hardware, run one sim-to-real perturbation test in MuJoCo or Isaac Lab: apply a uniform random shift of plus or minus 15 cm to object position and record how success rate degrades. If success drops below 50% at 10 cm shift, the demonstration dataset covers too small a workspace volume; add scripted or teleoperated data from the perimeter of the target workspace rather than retraining from scratch.

An end-to-end policy earns its keep only under a closed-loop contract that fixes how perception, estimation, planning, learning, and control assemble into a system. That contract names five things: the observation stream, the action representation, the timing budget, the safety boundary, and the result artifact. It is what turns a readable concept into a system a skeptical builder can test.

Separate the conceptual claim, the systems claim, and the evidence claim. A good explanation, a clean API, and one successful rollout are different kinds of evidence, and the section should keep them distinct.

A robust implementation starts with one inspectable baseline whose artifact records observations, actions, units, timestamps, seeds, termination reasons, and the perturbation applied. The maintained-tool version is useful only if it preserves that schema and lets the comparison remain construct-matched.

1. Write a one-paragraph task contract with observation, action, success, failure, and safety fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save one artifact containing configuration, seed, metrics, traces, and failure labels.
5. Compare methods only when the same script evaluates the same panel, split, seed set, and metric.

When End-to-end learned policy pipeline fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

For an end-to-end policy, the first isolation test is a nearest-neighbor audit of the training set: find the closest logged scenes, goals, and actions to the failed rollout. If the failed condition is absent, the diagnosis is coverage rather than architecture. If similar cases exist but the action scale, timing, or gripper convention differs, the diagnosis is representation alignment. If similar cases exist and conventions match, inspect the model's temporal window and action horizon before retraining.