Section 1.2: Why intelligence needs a world; the perception-action loop

"I sensed, therefore I acted. Then I had to sense what acting had done."

A Reflective Mobile Robot

A world supplies consequences, and consequences close the loop

Aim a thrown dart with your eyes closed and you can still compute a perfect release, yet the dart misses, because the one input you refused to use was the sight of where the last dart landed. That refusal is the whole difference this section is about, drawn in Figure 1.2A as a single cycle: a sensory signal from the world enters the agent, a decision is formed, an action changes the world, and the changed world is what the agent senses next. A static model is graded once and stops. An embodied agent issues a command into a world whose response it must then sense, and that response is what tells the agent whether the command did what it intended. Strip away the sensing of the consequence and you are left with open-loop control: the agent computes a plan from its initial knowledge and executes it blind. Open-loop control works only when the world matches the model exactly. The moment friction, payload, wind, a miscalibrated actuator, or another agent perturbs the system, the executed plan and the actual outcome diverge with nothing to pull them back together. A plan that cannot sense its own failure is not control; it is a wish.

Write the world as a transition \(s_{t+1} = f(s_t, a_t, d_t)\), where \(d_t\) is an unmodeled disturbance, and let the agent observe \(y_t = h(s_t) + n_t\) through a sensor with noise \(n_t\). Open-loop control chooses the whole action sequence \(a_0, a_1, \ldots\) ahead of time from \(s_0\) alone. Closed-loop (feedback) control instead computes each action from the latest measurement: \(a_t = \pi(y_0{:}t)\). The difference is the entire subject. Feedback is the only mechanism by which information about \(d_t\), which by definition was never in the model, can re-enter the controller; it enters through its effect on the measured signal \(y_t\).

The closed loop, formalized

Figure 1.2 lays out these three stages as a closed path, sense-and-estimate (\(o_t \to b_t\)), decide-and-act (\(a_t = K(r - y_t)\)), and world transition (\(s_{t+1}=f(s_t,a_t,d_t)\)), with the dashed arrow marking the fed-back measurement that rejects the disturbance. The loop is five recurring operations executed every cycle: sense (\(y_t = h(s_t)+n_t\)), estimate state or belief (\(b_t = \mathrm{update}(b_{t-1}, a_{t-1}, y_t)\)), decide (\(a_t = \pi(b_t)\)), act (apply \(a_t\) to the world), and observe the consequence (\(y_{t+1}\)), which feeds the next cycle. When the true state is hidden, the estimate step is a recursive Bayesian update,

$$b_{t+1}(s') \propto O(y_{t+1}\mid s')\sum_{s} T(s'\mid s, a_t)\, b_t(s),$$

The transition model \(T\) pushes the belief forward through the action, and the observation model \(O\) reweights it by the new evidence. The prediction term makes a probing action valuable before it reaches the goal: an action that sharpens \(b_{t+1}\) pays off even when it makes no direct task progress. Consider the scale of that payoff. In practice, on partially observed navigation benchmarks of this kind, a policy that never probes (always acting greedily toward the goal) typically needs on the order of tens of thousands of training episodes to converge, while a policy that occasionally takes an information-gathering detour typically converges roughly two orders of magnitude faster, because each probing step collapses uncertainty that the greedy policy must otherwise resolve by stumbling through failure after failure. Partial observability is the normal case, not an exception. Cameras do not see behind objects, lidar misses glass, tactile sensors report contact only after contact, and a language instruction omits the operational state entirely. An agent that assumes full observability builds a controller for a world it is not in.

Disturbance rejection in one scalar plant

The cleanest instance is a regulator holding a measured signal \(y_t\) at a reference \(r\). The proportional feedback law is

$$u_t = K\,(r - y_t),$$

where \(e_t = r - y_t\) is the tracked error and \(K\) is the gain. Consider a scalar plant \(y_{t+1} = a\,y_t + b\,u_t + d\) with a constant disturbance \(d\). Under open-loop control with a precomputed \(u\), the steady state settles at \(y_\infty = (b\,u + d)/(1-a)\), so the disturbance \(d\) appears undiminished in the output: there is no term that cancels it. Under proportional feedback the closed-loop steady state is

$$y_\infty = \frac{b\,K\,r + d}{\,1 - a + b\,K\,}.$$

As the loop gain \(bK\) grows, where the loop gain is the product of the plant input sensitivity \(b\) and the controller gain \(K\), measuring how strongly a unit of error is amplified once around the loop, \(y_\infty \to r\) and the contribution of \(d\) shrinks like \(1/(bK)\). That \(1/(\text{loop gain})\) shrinkage is the mathematical content of disturbance rejection, and it is unavailable to any open-loop scheme because no precomputed sequence can reference a disturbance it never saw. To see the gap in a single number: with \(a=0.8\), \(b=0.5\), \(d=0.3\), and \(K=1\), the open-loop steady-state error is 1.50 while the feedback error is 0.14, a ten-fold reduction from a proportional law with no knowledge of \(d\) whatsoever.

The cybernetic lineage

None of the machinery in the regulator above, the error term, the disturbance rejection, the gain that trades speed against stability, is new; it was named and proved decades before the first deep network. The loop was formalized before modern AI existed. Wiener's Cybernetics (1948) named feedback as the common principle of control and communication in the animal and the machine, and made the point, central to this book, that purposive behavior is negative feedback from its own results, not a plan executed forward from initial conditions. Ashby's Introduction to Cybernetics (1956) gave the loop a hard quantitative constraint, the law of requisite variety: a regulator can hold a system's outcome within a target set only if the regulator commands at least as much variety (in the information-theoretic sense) as the disturbances it must counter. Stated compactly, with \(H\) for entropy, the residual variety of the outcome is bounded below by

$$H(\text{outcome}) \ge H(\text{disturbance}) - H(\text{regulator}),$$

so under-actuated or under-sensing controllers cannot regulate a high-variety environment no matter how the policy is tuned.

Powers' perceptual control theory (1973) inverted the usual reading of the loop. An organism does not control its output; it controls its perception, acting to bring a sensed variable to an internal reference. This reframing matters for physical robots because it drops the assumption that anyone can precompute the correct output. The agent keeps acting until its sensed signal matches the reference, and it absorbs mechanical compliance, joint flex, and surface irregularities without naming them as disturbances. A hierarchy of reference signals passed downward supplies the mechanism. A high-level goal sets the reference for a mid-level perceptual variable, which sets the reference for a lower-level one, down to raw joint torques. Each level corrects only its own sensed error, so higher levels stay insulated from low-level physical noise that the body's inner loops already suppress. All three threads converge on the same object that control theory states most operationally: a controller drives the difference between a reference signal \(r\) and a measured signal \(y_t\) toward zero. Brooks' "Intelligence without representation" (1991) pushed the lineage to its limit, arguing that tight perception-action loops coupled directly to the world can produce competent behavior with little or no central model at all.

Open-loop fails, feedback corrects: a runnable demo

Wiener, Ashby, and Powers each argued in words that feedback rejects what no plan ever saw; the cleanest way to believe it is to watch the two strategies diverge on a plant you can run in thirty lines.

Before reading on, guess: by how much does proportional feedback reduce the steady-state error caused by an unmodeled disturbance, compared to open-loop control on the same plant? In informal polling, engineers seeing this setup for the first time typically guess a 2x to 3x improvement. The answer, for even a modest gain, is typically closer to 10x, and the math below traces the reason: the disturbance's contribution shrinks like \(1/(\text{loop gain})\).

The script regulates a scalar plant \(y_{t+1} = a\,y_t + b\,u_t + d\) toward a setpoint \(r\), with a constant disturbance \(d\) that neither controller was told about. The open-loop controller computes the single command that would hit \(r\) for the modeled plant and applies it forever. The feedback controller ignores \(d\) entirely and acts on the measured error \(r - y_t\) each step.

# Setpoint regulation under an unmodeled disturbance.
# Open-loop applies a precomputed command; feedback acts on the measured error r - y.
a, b = 0.8, 0.5          # scalar plant: y_next = a*y + b*u + d
r = 1.0                  # setpoint (reference signal)
d = 0.3                  # disturbance the controllers were NOT told about
K = 1.0                  # proportional feedback gain (keeps pole a - b*K = 0.3 stable)

# Open-loop command assumes d == 0: solve r = a*r + b*u  ->  u = r*(1-a)/b
u_open = r * (1 - a) / b

def simulate(feedback, steps=40):
    y = 0.0
    for _ in range(steps):
        u = K * (r - y) if feedback else u_open   # closed loop vs fixed command
        y = a * y + b * u + d                     # disturbance enters every step
    return y

y_open = simulate(feedback=False)
y_fb = simulate(feedback=True)
print(f"open-loop steady-state y = {y_open:.4f}  (error {r - y_open:+.4f})")
print(f"feedback  steady-state y = {y_fb:.4f}  (error {r - y_fb:+.4f})")
print(f"feedback rejects {100 * (1 - abs(r - y_fb) / abs(r - y_open)):.1f}% of the open-loop error")