Section 58.5: What is still unsolved (long-horizon reasoning, reliability, real-world RL)

"I can solve the task once. Reliability is the part where the task notices."

A Long-Horizon Policy After Minute Twelve

A policy that nails a grasp 97 times out of 100 sounds nearly solved, yet chain forty of those grasps into one twenty-minute task and the same policy succeeds barely three times out of ten: that collapse, hiding in plain sight behind an impressive single-step number, is the territory this section maps. To make it usable we define the object of study, connect it to the agent loop, then test it with a compact implementation.

These three problems persist not because researchers have ignored them but because each one attacks a different structural assumption that most learning pipelines take for granted. Long-horizon reasoning breaks the assumption that a good local policy composes into a good global trajectory. Reliability breaks the assumption that a held-out test set predicts deployment performance. Real-world RL (reinforcement learning) breaks the assumption that the number of training interactions is cheap. A household robot asked to tidy a kitchen, retrieve a specific medication, and then report back has to handle all three simultaneously: it must commit to a multi-step plan, stay safe around humans and fragile objects, and do so after only hours of on-hardware experience rather than millions of simulated steps. The diagram below locates each of the three gaps on the agent pipeline, showing exactly which inter-module link each one breaks.

The key question is practical: what must the agent know, what can it observe, what action is available, and what evidence shows that the action worked under the stated conditions?

Theory

Each of the three open problems attacks a different physical bottleneck. Long-horizon reasoning fails when subgoal memory decays faster than the task horizon. A Transformer-based policy such as RT-2 has a fixed context window. A kitchen-tidying task spanning twelve minutes and forty robot-steps silently drops the earliest observations. The robot then re-grasps objects it already placed or re-enters a room it already cleared. Re-grasping wastes irreversible time, risks collision with placed objects, and erodes the trust of any human co-worker. Unlike a silent software patch, a redundant physical action leaves a visible trace in the world. The compounding mechanism is direct: each dropped observation removes a constraint. Without memory that a cup is stowed, the agent treats the cup as unconstrained. Its value function assigns positive utility to grasping the cup again, producing a locally correct but globally redundant action at every subsequent step.

If memory decay corrupts what the agent remembers about its own past actions, the next bottleneck corrupts what the agent perceives about a world that has quietly moved on from the training set.

Reliability fails when the held-out test distribution does not capture deployment distribution shift. A policy trained in one kitchen typically generalises poorly in three common cases. A chair moves 30 cm. New lighting pushes RGB values outside the training range. A human body partially occludes a target object. Real-world RL fails because on-hardware episodes cost far more than simulated ones. Every Franka Panda grasp that ends in a drop risks joint-torque overload, finger wear, and an operator reset that costs two to five minutes of wall-clock time. That expense typically shrinks the effective sample budget by an order of magnitude or more below the millions of steps that sim-only benchmarks assume, though the exact factor depends on the task and hardware. A sim policy trains on 50,000 rollouts overnight at zero cost, while a real manipulator might safely complete 300 episodes before cumulative joint wear triggers a maintenance stop. A policy that works in simulation but fails on hardware is not a deployed policy; it is a hypothesis waiting for a robot to test it.

Worked Example

Consider a mobile manipulator asked to retrieve a specific medication from a cabinet and bring it to a person seated in an adjacent room. At minute zero, the task plan is correct. At minute four, a door swings shut and the robot's occupancy map goes stale. At minute eight, the medication bottle is partially occluded by another object. At minute eleven, the robot must recover from a failed grasp without any reward signal having been received yet. Each minute exposes a different failure: localization drift, sensor occlusion, and sparse delayed reward are all active simultaneously. A policy that scores 90% success on isolated 90-second pick-and-place trials in simulation may complete this task 0% of the time on first deployment, because none of the three failure modes appear in a 90-second sim episode.

Practical Recipe

1. Write the observation, action, and success metric before choosing a model.
2. Build a baseline that is simple enough to debug by inspection.
3. Add the library implementation only after the baseline behavior is understood.
4. Record failures as structured cases: perception error, state error, planning error, control error, or evaluation error.
5. Run at least one perturbation test before trusting the result.

def validate_ledger(payload: dict[str, object]) -> dict[str, object]:
    assert payload, "payload must not be empty"
    return payload

# Reliability ledger for an open-problem study.
ledger = {
    "episode_minutes": [5, 10, 20],
    "reliability": [0.84, 0.61, 0.33],
    "dominant_failure": "memory stale after interrupted subgoal",
    "interventions_per_hour": 2.4,
}
print(validate_ledger(ledger))

{'episode_minutes': [5, 10, 20], 'reliability': [0.84, 0.61, 0.33], 'dominant_failure': 'memory stale after interrupted subgoal', 'interventions_per_hour': 2.4}