Section 19.4: Exploration under partial observability

"The map had blank corners not because the robot had not moved there, but because the sensor could not see around the wall."

A Partially Observable Navigator

This section builds on the cost-gated probe framework introduced in Section 19.1; see that section for the full treatment of cross-chapter dependencies on reward specification, partial observability, and sim-to-real transfer. Here the focus narrows to the belief-state machinery required when observations alias multiple hidden states.

Two camera frames can be byte-for-byte identical and yet come from rooms a hundred meters apart. The deepest question in embodied exploration is therefore not "where do I go next?" but "do I actually know where I am?" The object of study is the belief state: the agent's maintained distribution or memory over what might be true behind the current sensor reading. When two physically distinct locations produce identical frames, we call it the observation aliasing trap. To break out of it, the agent must seek a disambiguating cue rather than act on the raw observation alone. As Figure 19.4A illustrates, this turns exploration into a memory problem: the agent must decide whether it found a new place or merely forgot an old one. Figure 19.4B traces the full loop the rest of this section formalizes, from observation to belief update to the entropy check that gates active sensing.

The observation aliasing trap matters physically because a robot cannot rewind time. If a wheeled platform commits to a direction in a symmetric corridor and picks wrong, correcting that error costs real time, battery, and actuator wear. One Habitat-Lab symmetric-floor study makes the gap concrete. A memoryless policy needed roughly 4,000 episodes to reach 60% task success. A policy that resolved the ambiguity with a single active-sensing step before committing reached the same success rate in under 200 episodes. The disambiguation check is cheap, yet each wrong turn compounds quickly. In manipulation, acting on the wrong contact assumption can jam a gripper or damage a part. Unlike a simulated agent that restarts instantly, a physical robot accumulates these costs, so resolving ambiguity before acting is cheaper than recovering afterward.

So what should a robot do when two places look identical and backtracking is expensive? The answer changes everything about how exploration is designed.

From single estimate to distribution

The trap is resolved by treating the agent's current knowledge as a probability distribution over hidden states rather than a single state estimate. When the agent observes a landmark with a known likelihood per state, Bayes' rule shifts probability mass toward the states that best explain the observation. Entropy of this distribution drops. The agent acts on the goal only once entropy falls below a threshold, using active sensing to seek whichever observation most reduces remaining uncertainty.

The key question is practical: what hidden variable matters, what observation can disambiguate it, and what evidence shows that the agent explored to reduce uncertainty rather than wandering through aliased views?

When to Use Belief Tracking

Partial observability is worth the added complexity only in specific conditions. A memoryless policy suffices when the environment is fully observable. It also suffices when aliasing is rare enough that the agent can absorb occasional wrong actions, or when episodes are so short that history carries no useful signal. Memory and belief tracking become necessary in three situations. First, two or more distinct states produce identical observations for several consecutive steps. Second, acting under aliasing is costly, such as gripping the wrong face of an object and requiring a full reset. Third, the agent must return to previously visited locations and distinguish them from visually similar unvisited ones.

In practice, a simple diagnostic reveals which regime applies: compare a memoryless policy against a policy with a short recurrent window on a task panel that includes symmetric corridors or repeated object configurations. If performance is indistinguishable, the environment does not require belief tracking for that task. If the recurrent policy succeeds substantially more often, aliasing is consequential and the belief machinery in this section is warranted.

Theory

At time \(t\) the agent receives an observation \(o_t\), maintains a belief \(b_t(s)\) over hidden states, chooses an action \(a_t\), and updates the belief after observing \(o_{t+1}\). The update is the core move: exploration should choose actions that make important hidden states easier to distinguish.

This is the mathematical proof promised earlier: the worked example below shows entropy falling step by step (\(2.0 \to 1.51 \to 0.99\) bits) as each active-sensing observation is incorporated, giving a concrete, checkable instance of the general claim that a correctly chosen disambiguating action reduces belief entropy.

The practical design rule is to make aliasing explicit. Inputs, outputs, assumptions, timing, and failure modes should include the hidden variable, the memory horizon, the observation that resolves ambiguity, and the diagnostic that detects belief collapse.

The algorithm below uses two terms already introduced above: "frontier action" names the coverage-advancing move shown as the green branch of Figure 19.4B, and "belief-collapse" names the failure case where an active-sensing step fails to lower entropy as expected.

Worked Example

Code Fragment 19.4.1 shows a tiny belief update for two visually aliased corridors. When the belief remains uncertain, the agent chooses an information-gathering action rather than pretending the observation fully identifies the state.

# Maintain a belief over two hidden states that share similar observations.
# High entropy triggers an active-sensing action instead of blind movement.
import math

belief = {"left_corridor": 0.5, "right_corridor": 0.5}
landmark_likelihood = {"left_corridor": 0.85, "right_corridor": 0.20}

for state in belief:
    belief[state] *= landmark_likelihood[state]
normalizer = sum(belief.values())
belief = {state: value / normalizer for state, value in belief.items()}

entropy = -sum(value * math.log2(value) for value in belief.values())
action = "inspect landmark" if entropy > 0.7 else "move to frontier"
print({state: round(value, 2) for state, value in belief.items()})
print("entropy", round(entropy, 2), "action", action)

Expected output: the belief should move toward one hidden state while still reporting uncertainty. If the trace contains only the latest observation, the agent has no audit trail for partial observability.

Practical Recipe

1. Name the hidden variable before choosing a model.
2. Log the observation, belief or memory state, action, and disambiguating cue together.
3. Build a memory-free baseline before adding recurrence or belief tracking.
4. Record failures as structured cases: observation aliasing, memory loss, belief collapse, stale map, unsafe active sensing, or evaluation mismatch.
5. Run at least one perturbation test that hides or corrupts the disambiguating cue.

Belief tracking earns its keep only when tied to a closed-loop belief contract. That contract names the observation stream, hidden variable, memory representation, action representation, active-sensing move, and evaluation artifact. Without it, a recurrent policy can look capable while using history in a way nobody can diagnose.

The graduate-level habit is to separate three claims. The conceptual claim explains why memory or belief should help. The systems claim explains which state estimate changes before action. The evidence claim records whether aliasing errors fall under the same seed panel and perturbation suite.

Each of those three claims is exercised by a different tool, so the choice of simulator and library is itself part of separating them cleanly.

Start with a tiny, inspectable belief trace, then graduate to a recurrent learner or navigation simulator. The baseline logs observation, belief or memory state, action, the hidden-state label where simulation provides it, and the cue that resolved ambiguity. The library version emits the same schema, keeping the comparison same-task rather than a story stitched from separate experiments.

1. Write a one-paragraph belief contract with hidden state, observation, memory, action, success, and failure fields.
2. Start with the smallest simulator or wrapper that exposes aliased observations clearly.
3. Run one deterministic smoke test and one cue-corruption perturbation before scaling.
4. Save a single result artifact containing configuration, seed, belief traces, memory resets, metrics, and failure labels.
5. Compare methods only when one script evaluates memory-free and memory-enabled policies on the same task panel.

Even with that disciplined build process in place, runs will still fail, and the same artifact schema is what makes those failures diagnosable rather than mysterious. When partial-observability exploration fails, avoid labeling the whole method as weak. First assign the failure to observation aliasing, memory horizon, belief update, active-sensing choice, stale map, timing, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause.