Section 19.1: Why embodied exploration is expensive and risky

"Every observation the robot earns costs a movement it cannot take back."

Section 19.1

It builds on reward specification in Chapter 18: Reward Design and Goal Specification, reuses partial observability from Chapter 2: The Agent-Environment Interface, and prepares transfer testing in Chapter 20: Sim-to-Real Transfer.

Ask a simulated agent to explore and it can fail ten thousand times before lunch; ask a real robot the same question and the tenth failure may already have a technician on the floor and a cracked gripper on the bench. The central object that explains this gap is a probe: an action chosen mainly to reduce uncertainty rather than to complete the task immediately. In a simulator, a random-walk agent can execute 10,000 such probes in a training session at zero marginal cost; on a physical robot in a warehouse, that same number of unpriced probes would typically consume on the order of tens of hours of human reset labor and often risks hardware damage well before episode 500 in our experience with contact-rich manipulation pilots, which is the hidden tax on physical curiosity. In practice, a cost-gated policy that prices each probe before committing can often reach comparable behavioral competence in on the order of hundreds of physical episodes, while an unpriced curiosity policy typically needs one or two orders of magnitude more before the task-success curve stabilizes; teams commonly report exhausting their reset budget well before that point and declaring the method impractical.

Figure 19.1B shows how this accounting fits together: a candidate probe is split into information gain (positive) minus reset burden and hazard risk (negative), and the resulting net score must clear a budget gate before the probe is allowed to execute. The budget gate and the reset ledger it feeds are defined precisely in the Algorithm box below; for now, read the gate as a pass/reject check and the ledger as the running log of every probe executed, its score, and what it actually revealed. The key question is practical: what uncertainty does the probe reduce, what does it cost to execute, what does it cost to reset, and what evidence shows that the probe was worth taking?

A policy that works in simulation but destroys hardware on day one is not a policy: it is a liability waiting for a robot. Reset burden matters in embodied AI because the environment does not restore itself. In simulation a reset is a function call; on a physical robot, it is a person walking over, repositioning objects, and clearing faults. That asymmetry means two probes with identical information gain can differ by an order of magnitude in true cost. Any exploration policy that ignores reset burden overspends it until a human or a safety cutoff intervenes.

In practice, you estimate reset burden by timing manual resets during a pilot run, then fitting a linear model over object displacement, contact count, and floor area affected. The probe score multiplies the resulting scalar \(r(a)\) by \(\alpha_r\). Actions that push objects far from their nominal positions, or into hard-to-reach corners, then score lower even when their information gain is high.

Theory

Turning that intuition about reset burden and hazard into something a policy can optimize starts with writing the exploration loop and its cost streams down formally.

At time \(t\) the agent receives an observation \(o_t\), maintains an internal state estimate \(\hat s_t\), chooses an action \(a_t\), and observes a consequence \(o_{t+1}\). Exploration adds a second accounting stream: \(i(a_t)\) for expected information, \(c(a_t)\) for motion and time, \(r(a_t)\) for reset burden, and \(h(a_t)\) for hazard risk.

A useful embodied exploration objective is therefore not "maximize novelty." It is closer to choosing probes with high information per unit of recoverable cost, while refusing probes whose hazard or irreversibility exceeds the current safety envelope.

Worked Example

Code Fragment 19.1.1 scores three candidate probes by information gain, reset burden, and hazard risk. The numbers are small enough to inspect by hand, which is the point: before training a policy, a builder should know what the system treats as expensive.

# Price exploration probes by information gained and physical cost.
# This makes reset burden and hazard risk visible before training.
probes = [
    {"action": "open drawer", "info": 0.80, "reset": 0.70, "hazard": 0.20},
    {"action": "tap handle", "info": 0.45, "reset": 0.10, "hazard": 0.05},
    {"action": "drive behind shelf", "info": 0.90, "reset": 0.95, "hazard": 0.35},
]

for probe in probes:
    score = probe["info"] - 0.5 * probe["reset"] - 1.5 * probe["hazard"]
    print(probe["action"], round(score, 3))

Expected output: the printed trace should show which probe the cost model prefers and why. If the most informative action always wins, the diagnostic is missing the embodied part of embodied exploration.

Practical Recipe

1. Write the observation, action, reset procedure, and success metric before choosing a model.
2. Attach a cost to every probe: time, energy, wear, human reset effort, and safety margin consumed.
3. Separate reversible probes from irreversible or hard-to-reset probes.
4. Record failures as structured cases: perception error, state error, contact error, timing error, reset error, or evaluation error.
5. Run at least one perturbation test that changes reset difficulty before trusting the result.

A cost contract makes this idea usable. It names the observation stream, the state estimate, the action representation, the timing budget, the reset budget, and the evaluation artifact. Without it, a model looks capable in a notebook, then fails the first time a sensor drops a frame, a controller saturates, or an object cannot be restored.

Separating the four claims

A single sentence like "the probe reduces uncertainty" actually bundles four separate promises, and treating them as one is why cost-aware exploration claims are hard to audit. The graduate-level habit is to separate four claims. The conceptual claim explains why a probe should reduce uncertainty. The systems claim explains which interface it changes. The safety claim states which states must remain reachable. The evidence claim records which measurement would convince a skeptical builder.

Each of those four claims leans on a different piece of tooling to make it measurable, which is where the choice of simulator, dataset, and robot stack starts to matter.

A robust implementation starts with a tiny, inspectable reset ledger and only then moves to a maintained simulator or robot stack. The baseline should log inputs, outputs, units, timestamps, termination conditions, reset effort, and hazard flags. The library version should produce the same artifact schema, so the comparison is a same-task comparison rather than a story assembled from separate experiments.

1. Write a one-paragraph task contract with observation, action, success, reset, and failure fields.
2. Start with the smallest simulator, dataset, or wrapper that exposes reset burden faithfully.
3. Run one deterministic smoke test and one perturbation test that changes recoverability.
4. Save a single result artifact containing configuration, seed, metrics, reset counts, traces, and failure labels.
5. Compare methods only when one script evaluates them on the same task panel and reset budget.

When exploration fails, avoid labeling the whole method as weak. First assign the failure to perception, state estimation, planning, control, timing, reset, irreversibility, or evaluation. Then rerun one controlled perturbation that isolates the suspected cause.