Section 1.1: Static prediction vs. embodied interaction

"A classifier answers a question once. An embodied agent answers, then inherits the consequences."

Section 1.1

The formal object changes

Ask a static classifier the same question a thousand times and it gives the same answer to the same fixed test set; ask an embodied agent once, and its answer rewrites the next question it will ever see. That difference starts at the level of the formal object. A static predictor learns a function \(f(x)=y\). The input \(x\) is drawn from a fixed distribution \(\mathcal{D}\), the output is scored by a loss \(\ell(f(x),y)\), and the example ends. Crucially, \(\mathcal{D}\) does not depend on \(f\): the test set is the same whether the model is good or bad. A predictor can be wrong and remain indifferent to the consequences; an agent that is wrong inherits a world shaped by its mistake.

An embodied agent acts inside a controlled Markov process. At step \(t\) it receives an observation \(o_t\), maintains a belief or internal state \(b_t\) (introduced here as notation; the next section explains why it is needed and how it is updated), chooses an action \(a_t \sim \pi(\cdot \mid b_t)\), and the world transitions \(s_{t+1} \sim P(\cdot \mid s_t, a_t)\), emitting \(o_{t+1}\). The unit of analysis is the trajectory \(\tau = (o_0, a_0, o_1, a_1, \ldots, o_T)\), and the score is a functional of the policy,

$$J(\pi) = \mathbb{E}_{\tau \sim \pi}\!\left[\sum_{t=0}^{T} r_t - \lambda \sum_{t=0}^{T} c_t\right],$$

where \(r_t\) measures task progress, \(c_t\) measures cost such as collision risk, energy, or recovery effort, and \(\lambda\) encodes how much the evaluator penalizes unsafe or expensive behavior.

Why the agent carries a belief

The belief \(b_t\) exists because a physical robot's sensors never reveal the full world state. A depth camera gives a point cloud, not joint torques; an inertial measurement unit (IMU) gives acceleration, not foot contact forces. Collapse these partial signals into a single action without memory and you discard causal history the policy needs: whether a gripper is already loaded, whether a door was just pushed open, whether a prior slip shifted the object. Belief tracks that history so the policy can act on what the world is, not just what one sensor frame shows. Without it, a robot handling an occluded object would behave identically whether the object had moved one second ago or not.

Mechanically, \(b_{t+1} = \text{encode}(b_t, a_t, o_{t+1})\) fuses three inputs: the prior belief, the action just taken (which predicts the expected next state), and the new observation (which corrects for prediction error). A recurrent network hidden state, a Kalman filter covariance update, or a particle set reweight implements this, depending on how nonlinear the dynamics are. The key point is that the action \(a_t\) enters the update: the robot's own movement is evidence about where the world now is.

The decisive term is the subscript on the expectation: \(\tau \sim \pi\). The distribution of states the agent is judged on is produced by the agent. Improve the policy and you change the test set. This single coupling, absent from \(f(x)=y\), is the source of distribution shift, compounding error, exploration cost, and the need for closed-loop evaluation. The practical test for which formalism applies is whether the system's own output can change the distribution of its future inputs: a spam filter, an X-ray triage classifier, or a recommendation ranker scored on a fixed held-out set stays a static predictor even when deployed, because tomorrow's email or X-ray does not depend on today's classification; a robot arm, a drone, or a dialogue agent whose action shifts the next observation requires the closed-loop treatment developed in this section, regardless of whether it has a physical body. Figure 1.1 traces this coupling visually: an action's consequence becomes the next observation, so the evaluation distribution is generated by the very policy under evaluation. Figure 1.1A above shows the downstream effect, where a single prediction error stops at the output in a static model but propagates into a state change that the next observation inherits in the closed loop.

Why error compounds: the horizon penalty

The practical consequence of \(\tau \sim \pi\) is that imitation does not behave like supervised learning. Suppose a policy is trained by behavior cloning, where behavior cloning is supervised learning that maps observations directly to the expert's recorded actions, and makes a mistake with probability at most \(\epsilon\) on states drawn from the expert's distribution. In a static setting the expected number of mistakes over \(T\) examples is \(O(\epsilon T)\), linear and benign. In the closed loop, a single mistake moves the agent to a state the expert never visited, where the policy has no guarantee at all and is more likely to err again. Ross, Gordon, and Bagnell showed that this drives the expected cost of behavior cloning to \(O(\epsilon T^2)\), quadratic in the horizon. At a 2% per-step error rate, a 10-step horizon yields roughly 0.2 expected mistakes; a 100-step horizon yields roughly 20, ten times the linear prediction. Put concretely: in reported DAgger-style pick-and-place experiments, on-policy correction has typically reduced the demonstration count needed for reliable behavior by one to two orders of magnitude relative to pure behavior cloning, because the quadratic penalty is the dominant cost, not the per-step learning difficulty. The extra factor of \(T\) is the price of the feedback coupling: errors are not independent, they steer the agent into regions where further errors are likely.

A minimal demonstration of compounding

The \(O(\epsilon T^2)\) bound is easy to doubt, so the simulation below makes the quadratic penalty emerge from a coin flip per step. An agent stays on the expert's state distribution until a per-step error knocks it off; once off, it rarely recovers, because it was never trained on those states. Holding the per-step error budget fixed, we measure how off-distribution time grows with the horizon. Linear growth means errors are independent; super-linear growth is the embodied penalty.

# Compounding of behavior-cloning error in a closed loop.
# Holds per-step error fixed and varies the horizon; off-distribution time grows super-linearly.
import numpy as np

def rollout_off_distribution_steps(horizon, step_error_prob, recover_prob, rng):
    on_track = True          # the agent starts on the expert's state distribution
    off_steps = 0
    for _ in range(horizon):
        if on_track:
            if rng.random() < step_error_prob:   # a wrong action leaves the expert's distribution
                on_track = False
        else:
            off_steps += 1                        # time spent in states the policy never trained on
            if rng.random() < recover_prob:       # recovery is rare: no supervision off-distribution
                on_track = True
    return off_steps

rng = np.random.default_rng(0)
print(f"{'horizon':>8} {'off-dist steps':>16} {'steps / horizon':>16}")
for horizon in (10, 40, 160, 640):
    off = np.mean([
        rollout_off_distribution_steps(horizon, step_error_prob=0.02, recover_prob=0.05, rng=rng)
        for _ in range(20000)
    ])
    print(f"{horizon:>8} {off:>16.2f} {off / horizon:>16.3f}")

Where point accuracy misleads

Counting off-distribution steps tells us how badly error accumulates, but it says nothing about which errors actually sink a task; that question shifts attention from the frequency of mistakes to their location. Because the state distribution is policy-induced, where an error occurs matters more than how often. Consider a Franka Panda arm (a widely used 7-degree-of-freedom research manipulator with torque sensing at each joint) on a peg-in-hole task, where a peg-in-hole task requires the arm to align and insert a held part into a tight-clearance socket, so contact forces spike the instant the peg meets the hole. A 5 mm position error during free-space transit costs one corrective move. The identical 5 mm error at the moment of insertion contact bends the compliant joint into its force-torque safety limit, triggers an emergency stop, and requires a human reset. Two policies can share identical aggregate end-effector MSE on held-out demonstrations yet produce opposite closed-loop success rates, if one concentrates its residual errors near the contact phase. Legged platforms show the same asymmetry at step edges: Boston Dynamics Spot (a commercial quadruped robot) recovers from a misstep on flat ground in one gait cycle, but a depth-camera misread that places one foot 80 mm over a 150 mm ledge produces an irreversible tip. The design response is to evaluate on trajectories, to weight cost \(c_t\) by reversibility, and to prefer a lower-confidence visuomotor policy with a calibrated abstention threshold over a higher-accuracy model that acts confidently near irreversible contact boundaries.