Section 2.2: State, observation, hidden variables, partial observability

"My camera saw the block. My gripper discovered the friction. My log finally admitted both were incomplete."

A Probabilistic Robot Assistant

Hand a simulator the true pose, mass, friction, and contact state of every object and a robot will still grip air, because the policy never received any of it. State lives in the world; the agent acts on observation alone, its limited evidence. Figure 2.2 shows why that gap closes into a loop rather than a one-time deficit. This distinction is the vocabulary you need to debug almost every embodied AI system. A robot policy may see only pixels, joint encoders, force readings, and a stale command queue. That difference is not a nuisance. It is the problem. Figure 2.2A makes this concrete: the true world state is a full room, but the agent sees only the lit region, and the hidden variables sit in the dark, connecting the unseen state to future observations. A sensor suite that cannot measure friction, mass, or intent is not a modest limitation: it is a permanent negotiation between what the world is and what the agent is allowed to know.

State is the information sufficient to predict future dynamics and reward when paired with an action. Observation is the sensor-facing evidence available to the agent. Hidden variables are state variables that matter but are not directly observed. Partial observability is the gap that breaks policies in deployment: the normal condition in which the latest observation is not enough.

Theory

Let \(s_t\) be the world state and \(o_t\) be the observation emitted by sensors or an environment wrapper (the code layer, such as a Gymnasium wrapper, that packages raw sensor data into the fixed-format array a policy consumes). In a fully observable task, \(o_t\) contains enough information to act as \(s_t\). In embodied AI, that is usually false. Occlusion hides objects, contact reveals only local forces, latency makes images stale, and human intent is not directly measurable.

Hidden variables matter because physical systems couple visible and invisible quantities. Object mass, surface friction, and motor temperature each change how forces propagate through a mechanism, yet none of them appear in a camera frame. Ignoring them forces a policy to average over the worst-case distribution at runtime. That average collapses grasps on wet surfaces and stalls joints at thermal limits. The training cost is proportionally steep in representative contact-rich manipulation benchmarks. A policy given true friction as privileged state typically converges in roughly 300 episodes on a contact-rich task; the exact ratio varies by task and simulator, but the same architecture, denied that hidden variable, commonly needs upward of 50,000 episodes to match it, because it must re-discover friction from force variance on every new object. A robot that cannot model a hidden variable cannot predict when its own actions will fail.

The mechanism is indirect inference: a hidden variable leaves a signature in multiple sensor streams over time. Slip risk is not visible but it raises force-torque variance and shifts contact duration. By tracking these secondary signals across steps and weighting them through a likelihood model (a function scoring how probable the observed signals would be under each candidate value of the hidden variable), the agent reconstructs a distribution over the hidden quantity without ever measuring it directly.

The agent therefore maintains an estimate \(\hat{s}_t\), formally a belief \(b_t\), a probability distribution over possible states rather than a single guess. The algorithm below refers to this distribution directly, so it is worth fixing the notation here before working through the update steps: \(b_t(s)\) is the probability the belief assigns to state \(s\) at time \(t\), and \(\hat{s}_t = \arg\max_s b_t(s)\) is just the single most likely state under that distribution. That estimate may be a Kalman filter state, a particle filter, a learned recurrent state, a transformer memory, or a world-model latent (a compressed vector, learned rather than hand-designed, that a neural network updates each step to summarize everything it has inferred about the hidden state so far). The name matters less than the contract: what uncertainty does it represent, how is it updated, and how does action use it? The cost is measurable. In representative 2024 benchmarks, grasping policies given true object pose reach roughly 90% success in about 500 episodes; the same architecture on raw camera observations needs several thousand episodes to hit 85%, and still fails when lighting shifts. That order-of-magnitude gap is not a modeling failure. It is the price of partial observability.

Worked Example

Code Fragment 2.2.1 updates a tiny belief state from a partial observation. The hidden variable is slip risk, which the camera cannot directly see.

# Section 2.2: runnable checkpoint for State vs. observation.
# Keep the output small so the evidence record can be inspected directly.
belief = {"block_x": 0.50, "block_visible": True, "slip_risk": 0.20}
observation = {"detected_x": 0.54, "visible": True, "force_spike": False}

if observation["visible"]:
    belief["block_x"] = 0.7 * belief["block_x"] + 0.3 * observation["detected_x"]
else:
    belief["block_visible"] = False

if observation["force_spike"]:
    belief["slip_risk"] += 0.25
else:
    belief["slip_risk"] *= 0.95

print({"estimated_x": round(belief["block_x"], 3), "slip_risk": round(belief["slip_risk"], 3)})

Expected output: an updated position estimate and slip-risk estimate. The example should make clear that vision updates visible pose, while contact evidence updates a hidden physical variable.

Practical Recipe

1. List variables needed to predict dynamics, not just variables found in the sensor packet.
2. Mark each variable as observed, estimated, delayed, hidden, or evaluator-only.
3. Add uncertainty to every estimated variable and log that uncertainty.
4. Carry history when the current observation is insufficient.
5. Evaluate under occlusion, sensor delay, calibration drift, and contact changes.

Guarding Against Privileged-State Leakage

Naming the hidden variables is only half the discipline; the other half is making sure the agent is never accidentally handed the answers during training. State-observation discipline prevents privileged information leakage. In simulation the evaluator may know object mass, contact normal, true pose, and collision margin. The policy should receive only the observation channels that a deployed system can provide. A result that mixes those views may measure access to simulator internals rather than intelligence.

The practical artifact is a variable ledger. Each variable is marked as observed, estimated, delayed, hidden, or evaluator-only. The ledger should also name the sensor or estimator that produces it and the uncertainty attached to it.

Build a leak test before training. The test should compare evaluator state fields with policy observation fields and fail when evaluator-only variables appear in policy input.

1. List all variables needed to predict dynamics and reward.
2. Mark each variable as observed, estimated, delayed, hidden, or evaluator-only.
3. Attach units, uncertainty, and source sensor or estimator.
4. Check that evaluator-only variables are absent from policy input.
5. Stress the belief with occlusion, delay, calibration drift, and contact changes.

# Build a variable ledger and detect privileged-state leakage.
variables = {
    "true_pose": "evaluator_only",
    "rgb_crop": "observed",
    "last_seen_pose": "estimated",
    "slip_risk": "hidden",
    "contact_force": "observed",
}
policy_input = {"rgb_crop", "last_seen_pose", "true_pose"}

def privileged_leaks(variables: dict[str, str], policy_input: set[str]) -> list[str]:
    return sorted(
        name for name in policy_input
        if variables.get(name) == "evaluator_only"
    )

print(privileged_leaks(variables, policy_input))

With the ledger and leak test in place, the remaining work is diagnostic: when a clean setup still fails, you need a way to localize which part of the belief pipeline broke. When behavior fails under partial observability, classify whether the agent lacked a sensor, lacked history, carried stale belief, underestimated uncertainty, or received leaked training information. Each cause points to a different repair.