Section 3.4: Hybrid and hierarchical architectures

"Let the learned part handle what you cannot write down, and let the engineered part handle what you cannot afford to get wrong."

A Hybrid Systems Integrator

Ask a single neural network to plan a ten-second route and close a one-millisecond torque loop in the same forward pass, and it will fail at both: that contradiction is exactly what hybrid and hierarchical architectures exist to dissolve, by giving each timescale its own controller and a clean contract between them. Figure 3.4 makes the unifying point: every tier runs the identical closed-loop cycle (evidence in, decision out, consequence inherited by the next step), and only the clock speed changes from tier to tier.

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

Hybrid and hierarchical architectures exist because one controller rarely operates well at every time scale. Figure 3.4A shows the canonical layout: a high-level task planner issues subgoal commands to a mid-level skill selector, which delegates to low-level reactive controllers that close the loop at high frequency. A language-conditioned task policy may reason over minutes, a skill selector over seconds, and an impedance or velocity controller over milliseconds. The hierarchy separates "what should happen next" from "how this joint should move right now." A controller that must plan the next ten seconds and close a torque loop in one millisecond is not a controller: it is a contradiction.

Real systems make the motivation concrete. Boston Dynamics' Spot runs a three-tier hierarchy. A mission planner chooses navigation goals over tens of seconds. A locomotion controller adapts footfall patterns over roughly 0.5 seconds. Joint-level PD (Proportional-Derivative) controllers close the torque loop at 1 kHz. The frequency ratio across levels reaches 10,000 to one: 0.1 Hz at the top versus 1,000 Hz at the bottom. A flat architecture would force a single network to solve a 10-second navigation problem and a 1 ms torque problem in the same forward pass. That is why flat designs fail reliably at the extremes of either timescale. As a rough illustration of the sample-efficiency gap rather than a benchmarked figure, training a flat policy end-to-end typically needs on the order of tens of thousands of rollout episodes to cover both timescales, while a hierarchical design that trains each level separately can reach a comparable success rate in a small fraction of that, because each level sees a simpler, faster-converging problem.

SayCan (Ahn et al., 2022) places a large language model at the task tier and pairs it with a library of learned skills. Each skill carries a value function that estimates whether the robot can execute it now. The LLM proposes which skill to run; the value function acts as the initiation set \(I_\omega\) (the condition under which the option is allowed to start; defined formally below). RT-2 (Brohan et al., 2023) compresses some of this structure by fine-tuning a vision-language model to emit low-level actions directly. Even so, RT-2 typically runs inside an outer task loop that handles retries and subtask sequencing. No single model today operates reliably from natural-language intent down to 1 kHz torque output, which is why hierarchy persists in practice.

The option: a skill with start and stop conditions

These systems all gesture at the same underlying structure, a start condition, an autonomous skill, and a stopping rule, and that recurring shape is precisely what the formal option model makes explicit. The option view from hierarchical reinforcement learning gives a useful formal model. An option is a reusable skill with explicit start and stop conditions, written \(\omega = (I_\omega, \pi_\omega, \beta_\omega)\). Here \(I_\omega\) says when the skill may start, \(\pi_\omega\) maps local observations to low-level actions, and \(\beta_\omega\) says when the skill should terminate. The high-level policy chooses \(\omega_t\), while the low-level policy emits actions until termination. This architecture works under three conditions: skill boundaries are meaningful, termination is observable, and the high-level planner never asks a low-level skill to satisfy an impossible precondition.

If delegation means the low level owns the actuators until termination, then the moment of termination is where that ownership is handed back, which is why the termination condition deserves its own scrutiny. The termination condition \(\beta_\omega\) matters physically because a robot that never releases control of a completed skill either wastes actuator effort or damages the environment. A "reach" skill that keeps commanding motion after contact will jam a wrist joint or crush an object; a "pour" skill that does not stop once the cup is full wastes fluid and corrupts downstream tasks. On real hardware, undetected over-run also saturates motor current limits, tripping fault states that require manual recovery.

Mechanically, \(\beta_\omega\) is evaluated at every low-level timestep as a predicate over the current state: if \(\beta_\omega(s_t) = \text{true}\), control returns to the high-level policy; otherwise the low-level policy continues. In practice the predicate checks one or more sensor signals against a threshold, such as force exceeding a contact limit, joint angle reaching a setpoint within tolerance, or elapsed time exceeding a budget. The threshold choice is a tuning decision: too tight and sensor noise causes premature exit; too loose and the skill over-runs.

Worked Example

The option triple \(\omega = (I_\omega, \pi_\omega, \beta_\omega)\) becomes concrete when each part is a function. The example runs a two-level controller on a reach-then-grasp task: a high-level policy picks an option, the option's low-level policy emits millisecond actions, and the option's termination function decides when control returns upward. The trace logs every boundary, which is exactly the evidence a hierarchy needs to be debuggable.

import numpy as np

# State: (gripper_x, gripper_closed). Goal: reach object at x*=1.0 then grasp.
X_TARGET, REACH_TOL, STEP = 1.0, 0.05, 0.25   # STEP = move per low-level tick

class Option:
    def __init__(self, name, can_start, low_policy, done):
        self.name, self.can_start = name, can_start
        self.low_policy, self.done = low_policy, done

reach = Option(
    "reach",
    can_start=lambda s: abs(s[0] - X_TARGET) > REACH_TOL,   # I_omega
    low_policy=lambda s: (np.sign(X_TARGET - s[0]) * STEP, 0),  # pi_omega
    done=lambda s: abs(s[0] - X_TARGET) <= REACH_TOL)        # beta_omega

grasp = Option(
    "grasp",
    can_start=lambda s: abs(s[0] - X_TARGET) <= REACH_TOL,   # precondition!
    low_policy=lambda s: (0.0, 1),
    done=lambda s: s[1] == 1)

def high_level(s, options):
    for opt in options:
        if opt.can_start(s):
            return opt
    return None

s = np.array([0.0, 0.0])
for t in range(20):
    opt = high_level(s, [grasp, reach])      # try grasp first, fall back to reach
    if opt is None:
        print("no applicable option"); break
    dx, close = opt.low_policy(s)
    s = np.array([s[0] + dx, max(s[1], close)])
    print(f"t={t:2d} option={opt.name:5s} x={s[0]:.2f} "
          f"closed={s[1]} terminate={opt.done(s)}")
    if opt.name == "grasp" and opt.done(s):
        print("task complete"); break

Expected output: the controller runs reach until the gripper is within tolerance, the high level then switches to grasp because its precondition is finally satisfied, and the task completes. The characteristic hierarchy bug is a precondition violation: delete the can_start guard on grasp and it will try to close on empty space at x=0. The logged precondition and termination fields are what let you assign that failure to the right level instead of blaming the whole stack.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

A hybrid hierarchy earns its keep only under a closed-loop contract for how perception, estimation, planning, learning, and control fit into one system. That contract names five things: the observation stream, the action representation, the timing budget, the safety boundary, and the result artifact. It is what turns a readable concept into a system a skeptical builder can test.

Separate the conceptual claim, the systems claim, and the evidence claim. A good explanation, a clean API, and one successful rollout are different kinds of evidence, and the section should keep them distinct.

A robust implementation starts with one inspectable baseline whose artifact records observations, actions, units, timestamps, seeds, termination reasons, and the perturbation applied. The maintained-tool version is useful only if it preserves that schema and lets the comparison remain construct-matched.

1. Write a one-paragraph task contract with observation, action, success, failure, and safety fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save one artifact containing configuration, seed, metrics, traces, and failure labels.
5. Compare methods only when the same script evaluates the same panel, split, seed set, and metric.

Diagnosing Failures by Level

When Hybrid and hierarchical architectures fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

When your robot fails a task, how do you know whether to blame the planner, the skill, or the sensor? The answer almost always lives at a level boundary, not inside any single component.

The characteristic hierarchy failure is a mismatch between levels. The high-level planner may select "open drawer" when the gripper is not aligned, or the low-level skill may report success after moving the handle without opening the drawer. Diagnose this by logging skill preconditions, termination causes, and residual state error at every boundary. A hierarchy is healthy when each layer can say why it accepted the task and why it stopped.