Section 3.1: The canonical stack: sense, perceive, estimate, predict, plan, control, act

"Sense, perceive, estimate, predict, plan, control, act: seven words for the distance between a photon and a wheel turn."

A Stack Diagram With Arrows Pointing Down

A robot misses a grasp by two centimeters, and an engineer spends three days suspecting the planner before discovering the real culprit was a camera-frame label that perception attached two stages earlier: that is the cost of a stack you cannot read stage by stage. Reading the stack stage by stage means naming each of the seven stages, stating the typed input and output of every stage boundary, and using that boundary structure to localize a failure to a single stage. Figure 3.1 collapses those seven stages into the three roles (evidence, decision, consequence) and draws the closed feedback loop that ties the last action back to the next observation. Figure 3.1A lays out the full pipeline with each stage's input type, output type, and latency budget, and it is worth keeping in view as a reference while reading the rest of the section. The section first defines the object of study, then connects it to the agent loop, then tests it with a compact implementation.

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

The practical design rule is to make the interface inspectable before optimization begins: inputs, outputs, units, latency, bounds, and failure labels should all be visible in the saved artifact.

To see why that inspectability is non-negotiable, start with the single stage where most hardware-coupling assumptions hide: perception.

Before perception can do that work, the sense stage has to exist as its own boundary: sensing is the raw acquisition step, reading a physical transducer (camera, IMU, lidar, force sensor) at its native rate and native units, with no calibration, filtering, or semantic interpretation applied yet. That distinction matters in practice: two robots with the same perception model but different camera vendors need only re-derive the sense-to-perceive calibration, not rewrite the semantic layer, because the sense stage is where vendor-specific voltages and timing live and nowhere else.

The perceive stage converts that calibrated sensor data into objects a planner can reason about, such as obstacle polygons, surface normals, or object class labels. In a physical robot this separation is not aesthetic: raw sensor voltages carry device-specific units, distortions, and coordinate frames that vary by manufacturer. If a planner consumed raw voltages directly, replacing one camera model would require rewriting the planner. Keeping perception as a dedicated stage insulates every downstream component from hardware changes and lets the perception module be retrained independently when the sensor fleet changes.

Mechanically, the perceive stage applies a learned or hand-crafted function that maps calibrated data to a structured representation. A convolutional network maps an image tensor to a set of bounding boxes with class scores. A point-cloud filter removes returns outside the robot's reachable workspace and fits planes to the remainder. A stereo matching algorithm converts a rectified image pair to a dense disparity map. The output carries semantic or geometric labels and, critically, explicit uncertainty estimates. The downstream estimator consumes those estimates without inspecting sensor internals.

Every Arrow Is a Contract

What happens when the perception stage hands a bounding box in camera coordinates to a planner that expects world coordinates? The robot moves, the box does not follow, and the failure looks exactly like a planning bug until someone checks the frame label at the handoff. Every arrow in the stack below (named explicitly in the equation just after this paragraph) is a contract: violate the coordinate frame or unit convention at any one arrow and the bug hides inside a different stage entirely.

Because each of those arrows is a contract, it helps to write the whole chain out as one expression so every handoff has a name and a type. The canonical stack is useful because it gives each transformation a job that can be tested independently without forgetting the closed loop. A compact version is:

$$o_t \xrightarrow{\text{sense}} x_t \xrightarrow{\text{perceive}} y_t \xrightarrow{\text{estimate}} \hat{s}_t \xrightarrow{\text{predict}} \hat{s}_{t+1:t+H} \xrightarrow{\text{plan}} \tau_t \xrightarrow{\text{control}} a_t \xrightarrow{\text{act}} o_{t+1}.$$

Here \(o_t\) is the raw observation, \(x_t\) is calibrated sensor data, \(y_t\) is a semantic or geometric percept, \(\hat{s}_t\) is the agent's best state estimate, \(\hat{s}_{t+1:t+H}\) is the predicted future over horizon \(H\), \(\tau_t\) is the chosen trajectory or skill, and \(a_t\) is the executable command. The important assumption is that each arrow preserves enough information for the next stage while reducing ambiguity for the decision. The stack breaks down when one stage silently changes units, coordinate frames, latency, uncertainty, or success criteria.

Stack Stages in a Real System

The equation and algorithm above are deliberately abstract; the same seven named handoffs appear, with real sensors and real units in place of the symbols, in the deployed system described next.

Boston Dynamics Spot navigating a construction site uses this stack with separated layers. The sensing stage reads an Inertial Measurement Unit (IMU), stereo cameras, and lidar at up to 100 Hz. The perception stage runs a learned terrain classifier that labels ground patches as traversable or not, producing a 2.5-D cost map (a top-down grid where each cell stores a single height value plus a traversal cost, cheaper to search than a full 3-D point cloud) rather than raw point clouds. The estimation stage fuses wheel odometry with visual-inertial odometry to track the base pose. On flat concrete the estimate drifts less than 1 cm per 10 m of travel (as of 2023 factory calibration). On sloped gravel the uncertainty balloon grows large enough to trigger a re-localization request.

The predict stage models body dynamics over a 0.5-second horizon (the length of simulated future the planner rolls out before committing to a move) so the planner can avoid commands that would tip the robot before the next sensor update arrives. The plan stage uses model-predictive control to select a foothold sequence, and control converts those foothold targets to joint torques respecting per-joint limits of roughly 40 Nm. When the perception classifier mislabels a wet metal grate as traversable, the failure propagates silently through estimation and prediction until the control stage detects an unexpected ground-reaction force and requests a human override. This is a boundary failure: the fault originated at perception but was only observable two stages later. A stage that hides its assumptions is not a simplification; it is a deferred crash.

Worked Example

The cleanest way to make the canonical stack concrete is to give every arrow its own function and run one observation through all seven. The toy world below is a 1-D reaching task: the body is at position \(x\), a target sits at \(x^\star\), and the only actuator is a bounded velocity command. Each stage does its single job and hands a typed value to the next.

import numpy as np

rng = np.random.default_rng(0)
TARGET = 1.00          # x* the agent must reach (meters)
STEP_MAX = 0.20        # actuator saturation (max move per step, meters)

def sense(true_x):                       # o_t: noisy raw reading
    return true_x + rng.normal(0, 0.02)
def perceive(o):                         # x_t: calibrated, de-biased
    return o - 0.01                      # known sensor bias
def estimate(x_meas, x_prev, alpha=0.7): # s_hat: low-pass state filter
    return alpha * x_meas + (1 - alpha) * x_prev
def predict(s_hat, a_prev, H=3):         # roll the model forward H steps
    return s_hat + H * a_prev
def plan(s_hat):                         # tau_t: desired displacement to goal
    return TARGET - s_hat
def control(tau):                        # a_t: saturate to actuator limit
    return float(np.clip(tau, -STEP_MAX, STEP_MAX))
def act(true_x, a):                      # world transition -> o_{t+1}
    return true_x + a

true_x, s_hat, a = 0.0, 0.0, 0.0
for t in range(12):
    o      = sense(true_x)
    x      = perceive(o)
    s_hat  = estimate(x, s_hat)
    s_pred = predict(s_hat, a)
    tau    = plan(s_hat)
    a      = control(tau)
    true_x = act(true_x, a)
    print(f"t={t:2d} o={o:+.3f} s_hat={s_hat:+.3f} "
          f"pred={s_pred:+.3f} a={a:+.3f} x={true_x:+.3f}")

print(f"final error = {TARGET - true_x:+.3f} m")

Expected output: the agent drives the error toward zero in under ten steps, with the command \(a\) saturating at \(\text{STEP\_MAX}=0.20\) on the first moves and then shrinking as \(\hat{s}\) approaches the target. The value of the trace printed by Code Fragment 3.1.1 is that each column is one arrow in the equation above: if the robot misses, you can read off which stage first reported a wrong number. Try injecting a stale estimate (skip the estimate update for one step) and watch the error grow even though every other stage is correct.

Practical Recipe

1. Pin the physical interface first: specify sensor model and rate (e.g., Intel RealSense D435i at 30 Hz depth, Velodyne VLP-16 at 10 Hz lidar), actuator type and limits (e.g., Franka Panda joints capped at 87 Nm), and the closed-loop cycle budget before selecting any algorithm. A 50 ms budget shared across all seven stages leaves roughly 7 ms per stage on average; a stage that consumes 30 ms collapses the remaining six into a single tick.
2. Implement the simplest possible baseline for each stage independently and log every inter-stage handoff as a timestamped, frame-labeled tuple. A constant-velocity predictor and a proportional controller are enough to expose coordinate-frame mismatches before any learned component is introduced.
3. Replace baseline stages with learned components one at a time, keeping all other stages frozen. When swapping in a Convolutional Neural Network (CNN) terrain classifier trained on Open X-Embodiment or an RT-2-style VLA policy, where a VLA (Vision-Language-Action model) is a single network that maps camera images and a text instruction directly to robot actions, run the new component against the logged traces from the frozen baseline to confirm its output distribution is compatible with the downstream stage's input expectations.
4. Classify every failure at the stage boundary: perception misclassification (e.g., the wet-grate false-traversable case from Boston Dynamics Spot deployments), filter lag (state estimate covariance too small to catch abrupt terrain change), prediction horizon too short for the actuator's settling time, control saturation masking a planning error, or metric computed on a different coordinate frame than the controller expects.
5. Run at least one sim-to-real perturbation test: add 10 ms of artificial latency at the sensing-to-perception boundary in MuJoCo or Isaac Sim and verify that the control stage does not produce torque spikes above the actuator's rated limit. Latency sensitivity is the most common failure that passes simulation but degrades on hardware.

The canonical stack: sense, perceive, estimate, predict, plan, control, act becomes useful when it is tied to a closed-loop contract for how perception, estimation, planning, learning, and control are arranged into a system. The contract names the observation stream, the action representation, the timing budget, the safety boundary, and the result artifact. That is the bridge between a readable concept and a system a skeptical builder can test.

Separate three claims about the canonical stack: the conceptual claim, the systems claim, and the evidence claim. A clear explanation, a clean API, and one successful rollout are distinct kinds of evidence, and conflating them is how a working diagram gets mistaken for a working robot.

A robust implementation starts with one inspectable baseline whose artifact records observations, actions, units, timestamps, seeds, termination reasons, and the perturbation applied. The maintained-tool version is useful only if it preserves that schema and lets the comparison remain construct-matched.

1. Write a one-paragraph task contract with observation, action, success, failure, and safety fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes the task contract faithfully.
3. Run one deterministic smoke test and one perturbation test before scaling.
4. Save one artifact containing configuration, seed, metrics, traces, and failure labels.
5. Compare methods only when the same script evaluates the same panel, split, seed set, and metric.

When The canonical stack: sense, perceive, estimate, predict, plan, control, act fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, data coverage, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause. This pattern turns a disappointing rollout into a reusable diagnostic asset.

A practical diagnostic is to freeze the downstream stages and replay the upstream trace. If the controller succeeds when fed the logged plan, the control layer is probably not the first cause. If the planner succeeds when fed a corrected state estimate, the problem moves to perception or estimation. This replay habit turns the stack from a diagram into a fault isolation tool, and the difference is measurable: a blind search through seven integrated stages typically requires dozens of live runs, while staged replay pinpoints the offending boundary in one or two offline experiments.