"Each architecture fails in its own way. A modular pipeline breaks at the seams; an end-to-end policy fails everywhere at once and nowhere in particular."
A Post-Mortem With No Clear Owner
This section assumes familiarity with the four architecture types introduced in sections 3.2 through 3.6: the modular pipeline, end-to-end policy, hierarchical design, and dual-system design. The failure signatures catalogued here recur in Part V (section 19.3, safety and robustness) and Part VII (section 33.2, Large Language Model (LLM) planners as high-level controllers), where the same interface-mismatch and routing-threshold patterns appear at larger scale.
A warehouse robot using a modular pipeline stalls at a doorway because its object detector passes a bounding box that its planner cannot parse. A drone running an end-to-end policy crashes silently, with no component to blame and no log entry that isolates the fault. Both failures were predictable: each architecture has a characteristic weak point, and knowing those weak points in advance is what separates a robust deployment from an expensive field incident. As embodied AI moves into hospitals, roads, and homes, the cost of an unexpected failure mode climbs fast. The rest of this section maps the specific breakdowns that each architecture invites and identifies the diagnostic signals that reveal them before they reach a physical system. Figure 3.8A frames the payoff: because each design has a single characteristic weak point, knowing where it tends to break turns a mystery failure into a targeted fix.
Because the loop feeds each action back into the next observation, a fault injected at the decision stage reappears as corrupted evidence one step later, which is why failure symptoms surface several stages downstream of their true cause. This is the same diagram introduced as Figure 1.3.
Figure 3.8 makes the core reason concrete: because the control loop feeds each action back into the next observation, a fault injected at one stage resurfaces as corrupted evidence a step later, so symptoms drift downstream of their true cause. Give four engineers the same broken robot, one running a modular pipeline, one an end-to-end policy, one a hierarchy, one a dual-system design, and each will start their search in a completely different place, because each architecture buries its faults in a characteristic location. This section turns the four architecture types from sections 3.2 to 3.6 into a diagnostic playbook: it names the characteristic weak point of each design, then connects each weak point to a specific deployed system (the RT-2 action-token interface, the SayCan value-function precondition check, a ROS 2 pick-and-place transform chain), then compresses the whole procedure into a runnable oracle-substitution diagnostic.
The key question is operational: when a Franka arm misses a grasp or a warehouse AMR (Autonomous Mobile Robot, a wheeled base that navigates without fixed guide tracks) stalls at a doorway, which architectural boundary do you instrument first, and what single oracle substitution (corrected pose, corrected plan, corrected skill precondition, forced System 2 routing) flips the failed rollout back to success?
A representation earns its place when it changes the measurable action interface. In failure modes of each architecture, the reader should keep asking which decision becomes easier, safer, or more reliable.
Theory
The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.
That inspectable interface matters precisely because where you have to look depends on the design you chose. Failure analysis is architecture-specific because each architecture hides uncertainty in a different place. A modular stack exposes many interfaces but can lose performance through handoff errors. An end-to-end policy removes handoffs but hides internal causes. A hierarchy improves long-horizon structure but creates precondition and termination failures. A dual-system design adds a router, which can become the most important component in the system. A policy that works in simulation but collapses on hardware is not a validated policy: it is a hypothesis waiting for the real world to reject it.
| Architecture | Likely first suspect | Evidence to inspect | Best perturbation |
|---|---|---|---|
| Modular pipeline | Interface mismatch | frame, timestamp, covariance, message schema | Replay a corrected upstream message. |
| End-to-end policy | Data coverage or action convention | nearest training episodes, action scale, horizon | Hold the scene fixed and vary goal wording or initial pose. |
| Hierarchy | Skill precondition or termination | selected skill, precondition check, stop reason | Force the same skill with corrected preconditions. |
| Dual-system | Routing threshold | uncertainty, risk, selected path, deliberation time | Sweep uncertainty around the escalation threshold. |
A common assumption is that end-to-end policies eliminate interface mismatch failures by removing explicit module boundaries. That assumption is wrong. End-to-end policies move those boundaries inside the network weights. There, the boundaries become invisible to logging and cannot be patched individually. When an end-to-end policy fails, the perception-to-action interface still exists. It is now a learned mapping with no named checkpoint to inspect. Each architecture makes a different trade: modular pipelines make failures locatable but introduce handoff errors; end-to-end policies reduce handoff errors but make failures non-locatable without probing the latent activations directly.
In a modular pipeline, each stage has an explicit output contract: if localization is wrong, the log names the module, the timestamp, and the frame. In an end-to-end policy, the perception-to-action boundary still exists structurally, but it has been absorbed into learned weight matrices with no named checkpoint. When the policy fails, the symptom is visible at the action output, while the cause is distributed across layers with no direct correspondence to the stages in the failure-signature table. This is why the three-pass diagnostic for end-to-end policies relies on input perturbations and latent probing rather than stage-by-stage log inspection.
When diagnosing dual-system routing failures, set the escalation threshold using a held-out calibration split rather than tuning it on the training rollouts. In practice, sweep uncertainty_threshold across five evenly spaced values and pick the one that maximizes F1 on the calibration set; the default value shipped with most uncertainty estimators (e.g., MC-Dropout in PyTorch or the built-in conformal predictor in mapie) is calibrated for classification benchmarks, not robot action distributions, and will silently pass too many near-boundary inputs to the fast system. Log the router decision and its raw uncertainty score on every step so you can audit the boundary post-hoc without rerunning the full episode.
Interface mismatch is the characteristic failure of modular pipelines because each module was designed and tested in isolation, so the output contract of one module is only implicitly agreed upon by the next. In a physical robot, this matters immediately: a perception module might publish a 3-D pose in the camera frame at 30 Hz while the planner expects a world-frame pose at 10 Hz, and the mismatch produces a goal point that drifts with head motion rather than staying fixed in the world. The robot does not crash; it pursues a moving ghost.
The mechanism has one moving part. A module reads its input, transforms it, and writes its output to a shared message bus. If the downstream module expects a different coordinate frame, timestamp convention, or unit scale, it silently applies its transform to the wrong values. The error compounds at each handoff, so symptoms typically appear two or three stages after the cause. Without per-stage logging, this hides the true source. In one ROS 2 pick-and-place stack, a 2 cm position offset at the perception stage grew to a 14 cm gripper miss at the controller, because each of the three intervening transforms scaled the value before passing it on. Per-stage logging found this single bug in four minutes in that case; the symptom alone had taken three days to trace, though the exact ratio will vary with how many transforms sit between perception and control.
Checkpoint
So far: interface mismatch (a silent contract violation between two modules), compounding error (small offsets that grow at each handoff), and per-stage logging (recording inputs and outputs at every module boundary) are the three ideas that explain why a modular pipeline's failures hide between stages rather than at the stage that caused them.
Once per-stage logging has localized a compounding offset like the 14 cm gripper miss, the next question is what to do with that localization. The practical goal is not to produce a dramatic failure label. It is to produce the smallest intervention that flips the outcome while leaving the rest of the run unchanged. That intervention identifies the architectural boundary where the fix belongs.
When the log points nowhere
What happens when you have a failure and no log that points to a specific stage? In the RT-2 deployment (Brohan et al., 2023), a single action-token interface connects the vision-language backbone and the low-level controller. When the language instruction names an object absent from the training distribution, the backbone still produces a plausible token sequence. The action scale and gripper timing in those tokens are wrong for the actual scene. This pattern is typically not a perception error and not a control error, since neither module ever receives an out-of-range value; it is best described as an interface mismatch between the two subsystems, matching the first-suspect entry for an end-to-end policy in the table above. To identify it, log the raw action tokens alongside the controller commands, because the symptom (missed grasp) appears three steps after the cause (bad token at step 1).
The same diagnostic logic applies to hierarchical systems. In SayCan (Ahn et al., 2022), a high-level language planner selects skills whose affordance preconditions are checked against a value function, and failures cluster in cases where the value function is uncertain near the skill boundary, causing the planner to select a skill the robot cannot physically initiate from its current pose. The architecture differs but the diagnostic method is the same: isolate the boundary, substitute an oracle value, and check whether the failure disappears.
The mechanism in Failure modes of each architecture is the contract between representation and action. Name what enters the module, what leaves it, which assumptions make that transformation valid, and which log would reveal a bad handoff.
Worked Example
The section argues that a good failure analysis finds the smallest intervention that flips the outcome. The example makes that operational: a rollout depends on four stage outputs (pose, plan, skill boundary, routing), exactly one is corrupted, and a search over single oracle substitutions recovers which one. This is the three-pass diagnostic compressed into a few lines.
# A rollout succeeds only if every stage output is correct.
ORACLE = {"pose": "good", "plan": "good", "skill": "good", "route": "S2"}
def rollout(stages):
return all(stages[k] == ORACLE[k] for k in ORACLE)
# The failing episode: one corrupted stage (the planner chose a bad plan).
broken = {"pose": "good", "plan": "BAD", "skill": "good", "route": "S2"}
print("baseline success:", rollout(broken))
# Pass 2: try each single oracle substitution; report the minimal fix.
for k in ORACLE:
patched = dict(broken, **{k: ORACLE[k]})
if rollout(patched):
print(f"minimal fix: substitute oracle '{k}' -> success")
# Pass 3: a real cause should flip a PANEL, not one hand-picked case.
panel = [dict(broken), dict(broken, pose="BAD"), dict(broken, route="S1")]
flips = sum(rollout(dict(ep, plan="good")) for ep in panel)
print(f"'plan' oracle flips {flips}/{len(panel)} panel cases")
Step-Through: Single-Oracle Substitution Search
Trace the diagnostic with the failing episode broken = {pose: good, plan: BAD, skill: good, route: S2} against the oracle {pose: good, plan: good, skill: good, route: S2}. Baseline: rollout(broken) checks all four keys; plan is BAD while the oracle wants good, so all(...) returns False. Failure confirmed. Pass 2, substitute one stage at a time: patch pose to good (already good) gives {good, BAD, good, S2} still False; patch plan to good gives {good, good, good, S2} which is True (minimal fix found); patch skill still False; patch route still False. Only the plan substitution flips the outcome, so k* = plan. Pass 3, panel of three: apply the plan oracle to [broken, broken+poseBAD, broken+routeS1]. Episode 1 becomes all-good (flip, 1). Episode 2 still has pose = BAD so it stays False (no flip, 0). Episode 3 still has route = S1 so it stays False (no flip, 0). Flip count is 1/3: the plan oracle fixes only episodes whose sole defect was the plan, which is exactly the signature of a true single-stage cause rather than a coincidence.
Expected output: the baseline fails, the search identifies plan as the single substitution that restores success, and the panel check shows the plan oracle flips only the episodes whose sole defect was the plan. The discipline is the takeaway: a cause that flips one cherry-picked case is a clue, while a cause that flips a panel is evidence. The table above tells you which oracle to try first for each architecture, and this search tells you whether the guess was right.
The hand-built fragment is a visibility tool. Production work should move to maintained stacks such as Hugging Face Transformers, open Vision-Language Models (VLMs), OpenVLA (a Vision-Language-Action (VLA) model), openpi, LeRobot, and tool-calling planners once the section has made the interface, logging contract, and failure recovery path explicit.
Practical Recipe
- Write the observation, action, and success metric before choosing a model.
- Build a baseline that is simple enough to debug by inspection.
- Add the library implementation only after the baseline behavior is understood.
- Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
- Run at least one perturbation test before trusting the result.
The common mistake in Failure modes of each architecture is to celebrate the component score before checking the closed-loop handoff. The failure usually appears at the boundary: stale state, wrong frame, delayed action, saturated actuator, or metric that ignores the real task cost.
On a Franka Panda arm running a diffusion policy trained on LeRobot pick-and-place demonstrations, the team observed a 23% success drop when the table surface was changed from matte wood to glossy acrylic. Final-success logging alone showed only that grasp attempts failed; it gave no clue whether the fault was a depth-camera specular reflection corrupting the point cloud, the wrist-force sensor reporting near-zero contact force (because the glossy surface reflects differently under the structured-light pattern), or the diffusion policy sampling an action outside the gripper's joint-velocity limits. Once the team added per-step logs of the raw depth image, the 6-axis wrist-force reading, the sampled action vector with its clamping flag, and the gripper state machine transition, the root cause became unambiguous within two episodes: the depth camera was producing NaN voxels in the contact region, and the policy was receiving a zeroed-out observation that fell outside the training distribution of the Open X-Embodiment Franka subset. The fix (a structured-light exposure adjustment and a NaN-guard in the point-cloud pipeline) restored success to within 2% of the matte-surface baseline. Without the intermediate logs, the team would have incorrectly attributed the failure to the diffusion policy itself.
Real-World Application: Autonomous Driving
Waymo's Driver stack is modular by design, with separate perception, prediction, planning, and control modules connected by versioned message contracts; in practice this modularity is what lets engineers attribute a regression to one stage rather than to an opaque end-to-end policy. When a disengagement occurs, engineers replay the log and substitute a corrected upstream message (an oracle perception output) to confirm whether the planner or the perception module owns the fault, which is the same single-oracle-substitution diagnostic described in this section applied at fleet scale.
Architecture diagrams look tidy because they do not include the arrow labeled 'everyone assumed someone else checked that'.
Automated failure attribution for black-box policies. As VLA models such as OpenVLA (Kim et al., 2024, Stanford) grow into billions of parameters, the three-pass oracle-substitution diagnostic described in this section becomes impractical without automated causal probing. Current work (as of 2024) uses activation patching and causal tracing (adapted from mechanistic interpretability) to localize which attention heads encode corrupted scene state, allowing failures to be attributed to a specific layer range rather than an entire policy. The open challenge is that patching inside a transformer does not map cleanly onto the modular-pipeline stages in this section's table, so a new taxonomy of "soft boundaries" inside end-to-end policies is still missing.
Runtime anomaly detection without ground-truth labels. The GROOT and ReKep projects (2024, Stanford and UC Berkeley) embed lightweight observation-consistency monitors alongside deployed manipulation policies. Rather than waiting for task failure, the monitor flags distribution shift in real time using conformal prediction intervals over the policy's own latent embeddings. This extends the dual-system routing idea: instead of a fixed uncertainty threshold, the threshold is recalibrated online from a sliding window of recent rollouts. The remaining open problem is how to set the window length without ground-truth success labels during deployment, which is almost always the real-world condition.
Cross-architecture failure transfer. Work from the Physical Intelligence (pi) lab and from the RT-X collaboration (2024) shows that failure modes from one architecture type regularly predict failure modes in a transferred policy trained under a different architecture on the same task. The empirical pattern suggests that some failures are task-grounded rather than architecture-grounded, but no formal theory yet predicts which failure modes transfer and which are architecture-specific. A PhD student could build a systematic benchmark by training the same task under all four architecture types described in sections 3.2-3.6, applying the oracle-substitution diagnostic to each, and measuring the overlap in causal stage attributions across architectures.
Can you name the observation, state estimate, action, success metric, and most likely failure mode for failure modes of each architecture? If not, the system boundary is still too vague.
Failure analysis becomes useful once it is tied to a closed-loop contract: the observation stream, the action representation, the timing budget, the safety boundary, and the result artifact. That contract is the bridge between a readable concept and a system a skeptical builder can test.
Separate the conceptual claim, the systems claim, and the evidence claim. A good explanation, a clean API, and one successful rollout are different kinds of evidence, and the section should keep them distinct.
| Tool or Library | Role in This Topic | Builder Advice |
|---|---|---|
| ROS 2 | separates system modules while preserving message contracts and timing | Use it when the hand-built contract is clear and the experiment needs repeatable runs. |
| MuJoCo | gives architecture choices a repeatable simulated world for stress tests | Use it when the hand-built contract is clear and the experiment needs repeatable runs. |
| LeRobot | anchors modern policy architectures in reusable datasets and policy APIs | Use it when the hand-built contract is clear and the experiment needs repeatable runs. |
A robust implementation starts with one inspectable baseline whose artifact records observations, actions, units, timestamps, seeds, termination reasons, and the perturbation applied. The maintained-tool version is useful only if it preserves that schema and lets the comparison remain construct-matched.
- Write a one-paragraph task contract with observation, action, success, failure, and safety fields.
- Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
- Run one deterministic smoke test and one perturbation test before scaling.
- Save one artifact containing configuration, seed, metrics, traces, and failure labels.
- Compare methods only when the same script evaluates the same panel, split, seed set, and metric.
When Failure modes of each architecture fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.
Use a three-pass diagnostic. First, classify the architecture, because the likely hidden variable depends on the design. Second, replay the episode with one oracle substitution, such as corrected pose, corrected plan, corrected skill boundary, or forced System 2 routing. Third, rerun the same intervention across a small panel of failures. A cause that flips one hand-picked case is a clue; a cause that flips a panel is evidence. The payoff is concrete: in a modular warehouse stack, replacing a stale pose estimate with the oracle value restored task success in 47 of 50 failure episodes, while replacing any other single stage fixed fewer than 3, a ratio that made the interface mismatch unmistakable and cut debugging time from three days to four hours.
Algorithm: Three-Pass Architecture Failure Diagnosis
Input: failed episode \(\tau = (o_1, a_1, \ldots, o_T)\), architecture type \(\mathcal{A} \in \{\text{modular}, \text{end-to-end}, \text{hierarchical}, \text{dual-system}\}\), oracle stage outputs \(\hat{s}_k\) for each stage \(k\), panel of \(N\) similar failures \(\{\tau^{(i)}\}_{i=1}^{N}\)
Output: minimal causal stage \(k^*\), flip rate \(\rho \in [0,1]\) across panel, structured failure label \(\ell \in \{\text{perception}, \text{state}, \text{planning}, \text{control}, \text{routing}\}\)
- Pass 1 (Classify): identify architecture \(\mathcal{A}\) and select the prior suspect stage \(k_0\) from the failure-signature table: interface mismatch for modular, data coverage for end-to-end, precondition \(\phi(\theta_t)\) for hierarchical, routing threshold \(\alpha\) for dual-system.
- Pass 2 (Single oracle substitution): for each candidate stage \(k\), construct patched episode \(\tau_k\) by replacing stage \(k\) output with oracle value \(\hat{s}_k\) while holding all other stages fixed. Evaluate success \(R(\tau_k) \in \{0,1\}\).
- Find minimal fix: \(k^* = \arg\min_k \{|k| : R(\tau_k) = 1\}\). If no single substitution succeeds, record compound failure and proceed to step 6.
- Pass 3 (Panel generalization): apply oracle substitution \(k^*\) to all panel episodes: \(\rho = \frac{1}{N}\sum_{i=1}^{N} \mathbf{1}[R(\tau_{k^*}^{(i)}) = 1]\).
- Accept \(k^*\) as the causal stage if \(\rho \geq 0.6\); otherwise demote to a contributing factor and inspect the next-highest candidate.
- Assign failure label \(\ell\) based on \(k^*\): map interface mismatch to \(\ell = \text{planning}\), precondition violation \(\phi(\theta_t) = \text{false}\) to \(\ell = \text{control}\), routing error \(|\hat{\alpha} - \alpha| > \nabla\alpha\) to \(\ell = \text{routing}\).
- Log structured record \((k^*, \rho, \ell, \pi_{\text{fix}})\) where \(\pi_{\text{fix}}\) is the intervention applied, then flag for the smallest targeted fix at the identified boundary.
Hierarchical systems fail at preconditions because the world continues to change between the moment a skill is selected and the moment it begins executing. If the high-level planner chose "pick up the cup" when the cup was reachable, but the robot spent 400 ms navigating and the cup shifted, the skill's precondition is now false even though the plan was correct. Routing thresholds in dual-system designs fail for a different reason: they are calibrated on the training distribution, and near-boundary inputs (those where uncertainty is just below the escalation threshold) are exactly the inputs most likely to be out-of-distribution. The router passes these inputs to the fast reactive system with false confidence. Both failure modes worsen under time pressure, because the latency budget shrinks the window for detecting that the precondition has changed or that uncertainty has crossed the threshold.
Hands-On Lab: Build a Section Evidence Trace
Objective
Turn Failure modes of each architecture into a small artifact that compares a hand-built baseline with a maintained-tool shortcut under one perturbation. Be clear about what this artifact is and is not: it is a conceptual evidence-trace exercise built from synthetic records with NumPy and pandas, not a running embodied architecture, simulator, controller, or robot. Its purpose is to teach the comparison schema (contract, baseline, perturbation, postmortem) that you will later populate with real simulation runs in Part III and with hardware or realistic embodied-system evidence in Parts IX and XI.
What You'll Practice
- Define an observation, action, metric, and perturbation contract
- Build a minimal baseline trace
- Preserve the same schema for the library shortcut
- Write a failure postmortem from the evidence record
Setup
pip install numpy pandasSteps
Step 1: Define the contract
Write the fields that make two runs comparable.
Step 2: Record the baseline
Save one deterministic result before adding noise or latency.
Step 3: Add the shortcut
Run or sketch the maintained-tool version while keeping the artifact schema fixed.
Step 4: Apply one perturbation
Change exactly one condition and preserve the same logging fields.
Expected Output
The completed lab produces one table with baseline, shortcut, and perturbed rows, plus a short note explaining which comparison is valid because all metrics were co-computed under one schema.
Stretch Goals
- Add a second seed and report mean and spread.
- Write a one-paragraph postmortem that separates root cause from symptom.
Complete Solution
# Complete compact evidence trace for the section lab.
# Extend these records with values produced by your actual environment or simulator.
import pandas as pd
records = [
{"run": "baseline", "seed": 0, "success": 0.72, "failure_label": "none"},
{"run": "library_shortcut", "seed": 0, "success": 0.78, "failure_label": "none"},
{"run": "baseline_perturbed", "seed": 0, "success": 0.54, "failure_label": "latency"},
]
print(pd.DataFrame(records))Failure modes of each architecture is useful when it makes the perception-action loop more reliable, not when it merely adds a more impressive model name.
Project Ideas
Beginner (weekend): Build a two-module pipeline in Gymnasium (CartPole or LunarLander) where a hand-written observer module passes state estimates to a hand-written controller module through a dictionary interface; deliberately introduce a unit mismatch (radians vs degrees) and log which episode steps fail, practicing the Pass 2 oracle substitution from the diagnostic algorithm. The key challenge is making the interface contract explicit enough that a single wrong field name or scale factor produces a detectable, attributable failure rather than a silent performance drop.
Intermediate (1-2 weeks): In MuJoCo (using the dm_control suite or a PyBullet equivalent), train a hierarchical policy with two skill primitives (reach and grasp) using Gymnasium wrappers, then instrument the precondition check for each skill and log every case where the high-level planner selects a skill whose precondition is false at execution time; compare failure rates before and after adding a 200 ms precondition re-check before skill initiation. The key challenge is separating planning-time precondition truth from execution-time precondition truth across the latency gap, which requires timestamped state logs synchronized with the skill-selection event.
Design a method-matched experiment for Failure modes of each architecture. Specify the environment, observation schema, action interface, metric, and one perturbation that targets the section's core assumption.
What's Next?
Chapter 4 begins Part II by giving these systems a geometric language for space and motion.
Bibliography & Further Reading
Brohan, A. et al.. "RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Control." (2023). https://arxiv.org/abs/2307.15818
A central reference for locating VLM and VLA models in embodied control stacks.
Todorov, E., Erez, T., and Tassa, Y.. "MuJoCo: A physics engine for model-based control." (2012). https://mujoco.org/
A widely used simulator for architecture and control experiments.
Quigley, M. et al.. "ROS: an open-source Robot Operating System." (2009). https://www.ros.org/
The systems reference for modular robot software and message-passing architecture.