"A robot that thinks slowly and acts quickly still needs a treaty between the two governments."
A Systems Architect
This section builds on the System 1 / System 2 framing introduced in section 3.6 and the affordance representation covered in section 27.5; reviewing those two sections will sharpen the discussion of what belongs in the context packet. The dual-system interface concept recurs in section 51.3 alongside long-horizon planning, and section 35.4 extends the argument by asking how to evaluate whether these architectures actually generalize across embodiments.
A humanoid robot receives the instruction "hand me the red mug." Its vision-language backbone parses the scene, locates the mug, and decides on a grasp plan, all in a few hundred milliseconds. Meanwhile its wrist controller must close on a moving target at ten times that rate. Those two clocks cannot share a single policy without one of them breaking. GR00T N1.5, Helix, and Gemini Robotics all reached the same conclusion in 2024 and 2025: split the stack deliberately. By the end of this section you will be able to trace exactly what crosses the boundary between the slow and fast layers, why that interface contract determines whether generalization holds, and where each of these systems draws the line differently.
Why Split The System At All?
Picture a robot hand catching a cup that has just slipped off a shelf: the fingers must react in the time it takes you to blink, yet the decision to catch it at all came from a language model that thinks a hundred times slower. High-level language reasoning and low-level motor control live on different clocks. A policy that reasons over long-horizon context, open-vocabulary instructions, and spatial semantics may only need to update every 200-500 ms. A wrist controller closing on a moving object typically needs updates every 10-20 ms, roughly a 20x speed difference. Forcing one policy to handle both rates either starves the motor loop of fresh commands or overwhelms the semantic module with redundant re-plans on every tick. Figure 35.3A pictures this split as a two-story control room: an upstairs planner writing scene-level intentions and a downstairs motor room executing fast trajectories under tight timing constraints. To put that mismatch in concrete terms: a system that waits for a full semantic reasoning pass before issuing every motor command would need to slow wrist closure from 20 ms to 400 ms, long enough for a slipping cup to fall 80 cm before the fingers even begin to close. Dual-system Vision-Language-Action models (VLAs) acknowledge that asymmetry instead of forcing one module to do both jobs on the same schedule.
GR00T N1 and N1.5 explicitly present this design as a System 2 vision-language reasoning module feeding a System 1 diffusion action model. Helix and Gemini Robotics make related moves with different packaging: a richer semantic layer provides task or scene context, then a downstream motor policy converts that context into reactive motion. The details differ, but the shared systems idea is a boundary between semantic deliberation and motor execution.
The success of a dual-system model depends less on the slogan "reason then act" than on what exactly crosses the boundary: goals, waypoints, affordance maps, language tokens, latent plans, or action proposals.
A Timed Interface
Because that boundary is where the design lives, the first thing to pin down is not what the packet contains but when it refreshes, so it helps to write the two clocks down explicitly. A clean formalization uses a slow context variable \(c_k\) and a fast control loop:
$$c_k = g_\psi(o_{1:t_k}, q, h_{k-1}), \qquad a_t = \pi_\theta(x_t, c_k), \qquad t_k \leq t < t_{k+1}.$$
The slow module \(g_\psi\) updates at times \(t_k\) using accumulated observations, instructions, and history. The fast module \(\pi_\theta\) consumes the current state \(x_t\) and the latest context packet \(c_k\) on every control step until a new slow update arrives. This equation forces the designer to specify the refresh rate and the contents of \(c_k\). Figure 35.3B below diagrams this timed interface directly.
Algorithm: Dual-System Inference Loop
Input: observation history \(o_{1:t}\), natural-language instruction \(q\), slow-module parameters \(\psi\), fast-policy parameters \(\theta\), slow refresh period \(T_s\), total horizon \(T\)
Output: executed action sequence \(a_1, a_2, \ldots, a_T\); updated context history \(h_K\)
- Initialize history \(h_0 \leftarrow \emptyset\) and step counter \(t \leftarrow 0\).
- Compute the initial context packet: \(c_0 \leftarrow g_\psi(o_{1:0},\, q,\, h_0)\).
- For each slow epoch \(k = 0, 1, 2, \ldots\) with \(t_k = k \cdot T_s\):
- \(\quad\) Run the fast motor loop for steps \(t = t_k, \ldots, t_{k+1} - 1\):
- \(\quad\quad\) Observe current state \(x_t\); compute action \(a_t = \pi_\theta(x_t,\, c_k)\).
- \(\quad\quad\) Execute \(a_t\); if an abort trigger fires (perceptual novelty \(\delta_t > \delta_{\max}\)), break to step 7.
- \(\quad\) Append new observations to history: \(h_{k+1} \leftarrow h_k \cup \{(o_{t_k:t_{k+1}}, a_{t_k:t_{k+1}})\}\).
- \(\quad\) Refresh the context packet: \(c_{k+1} \leftarrow g_\psi(o_{1:t_{k+1}},\, q,\, h_{k+1})\).
- \(\quad\) If the task goal is satisfied or \(t \geq T\), terminate and return \((a_{1:t},\, h_{k+1})\).
- Increment \(t\) and \(k\); continue from step 3.
Code Fragment 1 makes that timing split concrete with a toy scheduler.
# Refresh the semantic plan every three control steps.
# The fast controller reuses the latest plan until a new one arrives.
observations = ["drawer closed", "drawer opening", "drawer open", "grasping mug", "lifting mug"]
latest_plan = None
for step, obs in enumerate(observations):
if step % 3 == 0:
latest_plan = f"plan@{step}: open then grasp"
print(f"step={step} obs={obs} using={latest_plan}")
step=0 obs=drawer closed using=plan@0: open then grasp step=1 obs=drawer opening using=plan@0: open then grasp step=2 obs=drawer open using=plan@0: open then grasp step=3 obs=grasping mug using=plan@3: open then grasp step=4 obs=lifting mug using=plan@3: open then grasp
The expected output is a trace where the fast motor loop reuses a still-valid plan until the world state crosses a semantic boundary and the slow planner refreshes at `plan@3`. If the planner refreshed on every step, the architecture would be semantically expressive but latency-heavy; if it never refreshed, the controller would drift on stale intent.
Step-Through: Dual-System Inference Loop
Trace the timed interface with concrete numbers. Set the slow refresh period \(T_s = 3\) control steps, total horizon \(T = 5\), and a novelty abort threshold \(\delta_{\max} = 0.7\). The robot is opening a drawer to grasp a mug.
Slow epoch k=0 (t=0): the slow module runs and emits context packet \(c_0 = \) "open then grasp", computed at wall-clock 0 ms.
Fast step t=0: state \(x_0 = \) drawer closed, novelty \(\delta_0 = 0.1 < 0.7\), action \(a_0 = \pi_\theta(x_0, c_0) = \) reach-to-handle. No abort.
Fast step t=1: state \(x_1 = \) drawer opening, \(\delta_1 = 0.2\), action \(a_1 = \) pull-handle. Still reusing the stale \(c_0\) because \(1 \bmod 3 \neq 0\).
Fast step t=2: state \(x_2 = \) drawer open, \(\delta_2 = 0.3\), action \(a_2 = \) release-handle. Slow module has not refreshed yet.
Slow epoch k=1 (t=3): \(3 \bmod 3 = 0\), so \(g_\psi\) refreshes using the new history and emits \(c_1 = \) "grasp mug" at wall-clock 600 ms (two 300 ms slow passes).
Fast step t=3: state \(x_3 = \) grasping mug, \(\delta_3 = 0.4\), action \(a_3 = \pi_\theta(x_3, c_1) = \) close-fingers. Now conditioned on the fresh packet.
Fast step t=4: state \(x_4 = \) lifting mug, \(\delta_4 = 0.5 < 0.7\), action \(a_4 = \) lift. Goal satisfied, \(t \geq T\), loop terminates. The fast loop ran 5 times; the slow module ran only twice, a 2.5x reduction in expensive semantic passes. Had a hand displaced the mug at t=3 (\(\delta_3 = 0.9 > 0.7\)), the abort would have fired, marked \(c_1\) invalid, and forced an early slow refresh before t=4.
Fixed-rate plan refresh (the step % N == 0 pattern) is rarely enough on its own. Wire a second refresh trigger keyed to a perceptual event flag, such as a grasp-contact boolean or a segmentation-confidence drop below a threshold, so the slow module also re-plans mid-chunk when the scene changes unexpectedly. In openpi-style serving stacks (openpi is Physical Intelligence's open-source runtime for serving flow-matching VLA policies), pass this flag as a side-channel alongside the observation dict rather than encoding it in the image tokens, which keeps the semantic model's input distribution clean. In a ROS 2 deployment, send a GoalHandle.abort() from the fast-loop node and immediately re-trigger the slow planner action rather than letting the executor silently reuse the stale goal, because a cancelled action leaves a clear log entry while a silently stale plan does not.
What belongs inside \(c_k\) is the context packet design problem. Each format carries a distinct tradeoff. A language summary ("move the cup to the tray") is easy to generate, but it gives the fast module no geometric anchor, so the module must re-derive grasp orientation from raw perception on every step. A 3-D waypoint or grasp pose is geometrically precise but goes stale the moment the object shifts. An affordance heatmap is richer but costs more to compute at slow-module rate. GR00T N1.5 uses a latent embedding that blends visual and language context, trading interpretability for compactness. Helix passes whole-body pose targets. Gemini Robotics-ER encodes spatial relations from an embodied-reasoning step. Richer context reduces fast-module guesswork, but it also increases bandwidth and staleness risk when the scene changes quickly.
Think of the context packet like a navigator's briefing to a driver before entering a tunnel with no phone signal. A one-sentence summary ("head north") is easy to hand over but leaves the driver guessing at every junction inside. A turn-by-turn printout is precise, but if road works have moved a cone since the printout was made, the driver will steer into it confidently. A live GPS feed is the richest option, yet it demands a data connection the tunnel cannot guarantee. The context packet designer faces exactly this tradeoff: brevity keeps the handoff cheap but forces the fast loop to improvise; precision removes guesswork but goes stale the moment the scene shifts.
A context packet that is too thin forces the fast loop to guess; a context packet that is too precise breaks the moment the world shifts by a centimeter.
The toy scheduler makes the timing issue visible in a dozen lines. In practice, openpi-style serving stacks, OpenVLA inference wrappers, ONNX Runtime or TensorRT deployment paths, and ROS 2 action servers are where you log plan refresh rate, action latency, and stale-context failures. The maintained stack handles batching, device placement, middleware timing, and runtime orchestration so the experimenter can inspect the semantic-to-motor handoff itself.
| Tool or stack | What it anchors | Why it matters here |
|---|---|---|
| openpi | Plan-to-action serving boundary | Useful for inspecting where semantic context is handed to a motor policy. |
| OpenVLA | Open VLA inference and adaptation path | Lets a lab test whether the slow semantic context actually improves downstream action selection. |
| ONNX Runtime or TensorRT | Low-latency deployment path | Critical when the fast loop has to stay real-time after a large semantic model is added. |
| ROS 2 actions | Typed execution interface with feedback | Useful for exposing cancelation, completion, and stale-plan interrupts explicitly. |
How Current Systems Differ
With the interface contract and its context-packet tradeoffs now in view, the differences between the frontier systems reduce to the same three questions applied to real hardware: how fast each module runs, what crosses the boundary, and what happens when the packet goes stale. One recurring term in the table below is impedance controller, meaning a low-level controller that regulates the relationship between motion and force rather than commanding position alone, which is why it can detect a contact spike and trigger an abort.
| System | Slow module role and rate | Fast module role and rate | Context packet contents | Physical consequence of design choice | Main caveat |
|---|---|---|---|---|---|
| GR00T N1 / N1.5 | Multimodal vision-language backbone (Eagle2-based, i.e. built on NVIDIA's Eagle2 vision-language model) reasons over wrist-camera and head-camera RGB at roughly 200-500 ms per plan; pre-trained on Open X-Embodiment and the GR00T synthetic motion dataset across 50+ robot embodiments | Flow-matching diffusion transformer (a generative model that produces an action sequence by iteratively denoising a noise sample toward a target trajectory, the same family of technique introduced for action chunking in section 22.4) generates 50 Hz joint-position targets for up to 30 degrees of freedom (DOF) humanoid arms and hands; runs on the onboard Jetson Thor SoC | Latent embedding concatenating visual token summaries and instruction tokens; no explicit waypoint, so the fast module must resolve grasp orientation from the embedding on each step | Removing the explicit waypoint shrinks the packet to kilobytes but forces the diffusion head to re-derive finger pose from a compressed latent every 20 ms; on Franka Panda baselines the team reported a measurable drop in finger-tip placement accuracy for thin objects compared with waypoint-conditioned baselines | Architecture is open-sourced (Isaac GR00T SDK), but most labs will still need to re-fine-tune on their own embodiment; N1.5 transfer claims are vendor-reported. |
| Helix (Figure AI) | Vision-language model running at roughly 3-5 Hz reasons over stereo head cameras and task language on the Figure 02 humanoid; provides whole-body pose targets covering 35 DOF | Low-level impedance controller closes at 200 Hz over finger, wrist, elbow, and hip joints; a separate balance estimator runs in parallel at 1 kHz to keep the 67 kg humanoid upright during upper-body manipulation | Explicit whole-body joint-angle targets passed as a 35-element float vector; geometric precision is high, but when a grasped object slips during transit the fast layer has no semantic channel to request a re-plan and must rely on an impedance spike exceeding a contact threshold to trigger abort | Passing explicit joint targets means a 5 cm object displacement during a handover causes the fingers to close 5 cm to the left of the actual object; the team's reported solution is a contact-force abort threshold of roughly 15 N that re-triggers the slow module within one slow epoch | Evidence is primarily vendor video and blog posts; Figure AI has not released model weights or an independent benchmark suite as of mid-2025. |
| Gemini Robotics / ER | Gemini 2.0-based embodied-reasoning module runs spatial relation extraction and object-state tracking at roughly 1-2 Hz; trained with ALOHA 2 demonstration data and supplemented by RT-2-style web pre-training | Lightweight action decoder specializes to the target robot's kinematics (ALOHA 2 bimanual arms, Apptronik Apollo humanoid) at 10-50 Hz depending on the task phase; an ER-style reasoning trace is cached between slow refreshes | Spatial relation descriptor ("the red cup is 8 cm to the left of the blue bowl, currently upright") plus an action-type token; richer than a pure language summary but cheaper than a full affordance heatmap; the descriptor goes stale if a human hand enters the scene between slow refreshes | Encoding spatial relations rather than pixel affordances typically cuts context-packet GPU memory by roughly 4x compared with passing a full semantic segmentation map (a back-of-envelope estimate, not a figure from the technical report), which is consistent with deployment on a single A100 serving both the ER module and the action decoder; the trade-off is that sub-centimeter position changes are lost in the descriptor quantization | Technical report (arXiv 2503.20020) is more detailed than Helix, but model weights and ALOHA 2 fine-tuning code are not publicly released; independent replication on non-Google hardware remains limited. |
Checkpoint
So far: GR00T N1.5 trades an explicit waypoint for a compact latent (cheaper but harder for the fast module to resolve), Helix trades that same waypoint problem the other way by passing explicit joint targets (precise but blind to mid-transit slip until a contact-force spike fires), and Gemini Robotics-ER lands in between with a spatial-relation descriptor; keep this ordering in mind heading into the failure modes below, since each one fails differently when its packet goes stale.
The most common failure in dual-system deployment is stale-context drift: the slow module issues a plan ("grasp the red cup on the left") and the fast loop executes it faithfully, but by the time the wrist arrives, a human has moved the cup. The fast controller has no way to detect the mismatch because it only sees proprioception (the robot's own sensed joint angles and forces, as distinct from external vision) and the current context packet, not the semantic intent behind it. The result is a confident, well-executed grasp on empty air. Systems without an explicit abort trigger keyed to perceptual novelty and runtime monitoring will reproduce this failure on every scene change that outpaces the planner's refresh rate.
An abort trigger matters because a robot on a stale plan cannot stop itself through normal control logic: the fast loop is doing exactly what it was asked. The only fallback is a hardware torque limit, which fires far too late to prevent a collision or a missed grasp. A dedicated abort trigger restores semantic awareness to the fast loop without forcing continuous slow-module polling.
Mechanically, the trigger monitors a scalar signal computed at fast-loop rate: contact force, depth-image novelty, or a grasp-success confidence score. When that signal crosses a threshold, the fast controller halts its current action chunk, marks the context packet invalid, and interrupts the slow module to request a fresh plan. The slow module re-runs perception and returns a new context packet before the fast loop resumes. This closes the feedback gap that stale-context drift exploits.
A common assumption is that the slow vision-language module is where the "real intelligence" lives, so upgrading it automatically improves the robot's behavior. This is wrong in embodied AI because the context packet is the actual bottleneck: a more capable slow module that passes a poorly specified context packet (for example, a vague language summary with no geometric anchor) cannot help the fast module resolve grasp orientation any better than a weaker slow module with the same underspecified packet. The correct mental model is that slow module capability and context packet design are co-equal constraints; improving either one while holding the other fixed yields diminishing returns, and the interface contract must be redesigned together with any significant change to either layer.
Closed vendor systems can be technologically important without yet being textbook-grade evidence for a specific claim. Separate architecture lessons from benchmark claims unless an independent evaluation artifact exists.
A humanoid sorting task may need a slow module to infer "the left bin is for fragile objects" from language and scene context, while the fast module handles balance, wrist orientation, and grasp closure at control rate. If a new object slips, the fast layer may have to abort before the planner ever refreshes. That abort path is part of the architecture, not a postscript.
Real-World Application: Warehouse Humanoid Manipulation
Figure AI's Helix runs exactly this split on the Figure 02 humanoid for logistics package handling: a 3-5 Hz vision-language module reasons about which item to pick and where it goes, while a 200 Hz impedance controller closes the fingers and a parallel 1 kHz balance estimator keeps the 67 kg robot upright during the reach. When a grasped package slips, the 15 N contact-force abort threshold re-triggers the slow module within one slow epoch, restoring semantic awareness to a fast loop that would otherwise keep squeezing empty air.
Dual-system VLAs are a little like a chef and a line cook sharing one kitchen. If the order tickets are late or vague, the fastest hands in the room still plate the wrong dish.
What exactly would you put inside the slow context packet for a drawer-opening task: a language summary, a grasp waypoint, an affordance heatmap, or a full action chunk? Defend your answer in one sentence.
Learned interface contracts. Rather than hand-designing what the context packet contains, recent work trains the boundary itself. Adaptive context representations (as explored in the Pi0 flow-matching work from Physical Intelligence, 2024) learn a latent that the fast policy can query asynchronously, letting the slow module update at variable rates without stalling the motor loop. The open question is how to regularize these latents so they remain interpretable and transferable across embodiments.
Triggered re-planning via world models. Instead of fixed-rate or contact-force abort, several 2025 systems couple the slow module to a learned predictive world model that anticipates divergence before it is physically detectable. UniSim (Google DeepMind, 2024) and related video-prediction approaches provide a draft rollout of the next few control steps; when the predicted trajectory diverges from the live state by more than a threshold, the slow module is interrupted early. This removes the one-slow-epoch latency penalty of contact-only abort triggers.
Cross-embodiment context packets. The GR00T N1.5 transfer results and the RoboCasa benchmark (Nasiriany et al., 2024) raise the question of whether a context packet trained on one morphology can be reused by a different one. Early evidence suggests spatial-relation descriptors transfer more reliably than latent embeddings tied to a specific encoder's token structure, but systematic comparisons across three or more embodiment families do not yet exist.
Open problem for PhD students: design an evaluation protocol that separates context-packet quality from fast-policy quality in a dual-system VLA. Today, a single aggregate success metric conflates the two, making it impossible to attribute a failure to a stale or underspecified packet versus inadequate motor generalization. A controlled ablation framework that holds the fast policy fixed while varying only the context packet representation, run on a public benchmark such as RoboCasa or Open X-Embodiment, would provide that separation and give the field a shared tool for comparing interface designs across systems.
Dual-system robot foundation models matter because they acknowledge the mismatch between semantic reasoning time and motor control time. Their true quality lies in the clarity, timing, and fail-safe behavior of the interface between those loops. As the "Main caveat" column of the comparison table shows, GR00T N1.5's transfer numbers are vendor-reported, Helix's evidence is primarily vendor video and blog posts, and Gemini Robotics' technical report has not yet been independently replicated on non-Google hardware; treat every architecture lesson in this section as separable from those unverified benchmark claims.
Design a dual-system interface for a humanoid kitchen task. Specify the slow update rate, the fast control rate, the contents of the context packet, and the abort rule when fast execution detects a mismatch with the planner's assumptions.
Lab: Measuring Stale-Context Drift
Goal: empirically map how a dual-system's success rate collapses as the slow refresh period grows, and locate the abort threshold that recovers it.
Tools needed: Python with gymnasium-robotics (the FetchReach-v2 or FetchPickAndPlace-v2 environment), NumPy, and Matplotlib. No GPU required; this runs on a laptop in roughly 20 minutes.
Setup: implement a rule-based slow module that emits a 3D goal waypoint every \(T_s\) control steps, and a simple proportional-derivative fast controller that tracks the latest waypoint each step. Inject scene change by perturbing the goal position by a random 3-8 cm offset at a random step during each episode (simulating a moved object).
What to vary: sweep the slow refresh period \(T_s \in \{1, 3, 5, 10, 20\}\) control steps over 100 episodes each. Then add a novelty abort trigger (fire when the measured goal-to-gripper error jumps by more than a threshold \(\delta_{\max}\)) and sweep \(\delta_{\max} \in \{0.02, 0.05, 0.10\}\) meters.
What to observe: plot success rate against \(T_s\) with and without the abort trigger. You should see success degrade steeply as \(T_s\) grows past the perturbation timescale, and the abort trigger flatten that curve, but only when \(\delta_{\max}\) is small enough to catch real shifts yet large enough to ignore normal tracking noise. Record the \(T_s\) at which the no-abort curve crosses 50 percent success; that is your concrete stale-context drift horizon for this task.
Project Ideas
Beginner (weekend): Build a dual-system scheduler in Python using the FetchReach-v2 environment from gymnasium-robotics where a rule-based slow module issues a 3D goal waypoint every N steps and a PD controller fast loop tracks it; instrument the timing to measure how stale-context drift degrades success rate as N increases. The key challenge is exposing the exact control step at which the stale waypoint causes the first missed grasp, so the abort trigger threshold becomes observable rather than guessed.
Intermediate (1-2 weeks): Implement a dual-system VLA for a tabletop pick-and-place task in MuJoCo (or Isaac Lab) using LeRobot's diffusion policy as the fast motor module and a small vision-language model (such as PaliGemma-3B via the transformers library) as the slow semantic module communicating through a language-summary context packet; then replace the language summary with a 3D grasp waypoint and compare final-placement accuracy on 50 trials. The key challenge is designing the abort trigger so that contact-force spikes detected by the fast module correctly interrupt a stale grasp plan before the slow module's next scheduled refresh, without generating false positives on normal grasp contact.
What's Next?
Section 35.4 turns from architecture to evidence by asking how large behavior models should actually be evaluated, especially when aggregate success can hide embodiment-specific failure.
Bjorck et al. (2025). "GR00T N1: An Open Foundation Model for Generalist Humanoid Robots."
The clearest open reference for the dual-system framing in humanoid foundation models.
Google DeepMind (2025). "Gemini Robotics: Bringing AI into the Physical World."
The main source for Gemini Robotics and Gemini Robotics-ER, including the embodied-reasoning framing used in this section.
NVIDIA Research. "GR00T N1.5."
Useful for current architecture and performance claims, with the usual caveat that it is an official report rather than an independent benchmark paper.
An important frontier-watch source for whole-body VLA design in humanoids.