Section 56.1: Why memory matters; short- vs. long-term

"I remembered the last ten seconds perfectly, then promoted the wrong ten seconds forever."

A Short-Term Buffer Seeking Tenure

Why Memory Enters The Loop

Close a cabinet door in front of a kitchen robot and its mug vanishes from every camera frame; a memoryless policy will now re-reach into empty air on each step, as if the mug had ceased to exist the instant it left view. This is the everyday cost of partial observability: objects leave the field of view, force signals lag behind contact, and goals often remain active far longer than a single observation window. A policy that depends only on the current frame is therefore often too myopic to act. As Figure 56.1A captures, memory earns its place only when stored information changes a later embodied decision; storage for its own sake is just overhead.

Physical robots face sharper partial observability than simulated agents, because the body occludes its own sensors: a closing gripper blocks the wrist camera, a mobile base shadows its rear lidar, and actuator noise corrupts state estimates with no reset button to restore ground truth. The robot cannot replay a missed observation. It must act on an incomplete picture or wait, and waiting burns energy and task time. A memoryless robot is not cautious but amnesiac, repeating the same wrong reach every time the object slips out of view.

Mechanically, the agent maintains the belief state, a probability distribution over all world states consistent with the observation history. Each new observation updates this distribution via Bayes' rule. Memory is the substrate that carries the distribution forward in time between observations, so that a single occluded frame does not collapse all prior information about object location or contact history.

The important distinction is between memory that helps the next action and memory that should influence a later task, a gap that spans five orders of magnitude in time: working memory operates over tens of milliseconds while long-term memory must remain valid for minutes to months. A good design preserves only the information that meaningfully changes future control or planning.

Theory

The admission test becomes precise once you locate where each kind of memory enters the equations of control.

Where each memory tier sits in the control model

A compact memory-augmented control model is

$$h_t = f_\theta(h_{t-1}, o_t), \qquad m_t = \operatorname{Retrieve}(q_t, \mathcal M), \qquad a_t \sim \pi_\phi(a \mid o_t, h_t, m_t, g_t).$$

Figure 56.1B maps each symbol below onto a box in the loop: the latent state is the Working Memory box, the external store is the Long-Term Memory box, and the dashed arc is the promotion step that moves selected events from one to the other.

The latent state \(h_t\) is working memory: a compact representation of the recent past tuned for immediate control. The external store \(\mathcal M\) is long-term memory, tuned for persistence, indexing, and selective retrieval. The query \(q_t\) depends on task and context. Long-term memory is therefore not merely "more history"; it is a searchable support for future decisions.

Worked Example

The diagram's two tiers stop being abstract the moment a single grasp depends on both of them at once.

Suppose a kitchen robot loses sight of a mug when a cabinet door closes. Without working memory the grasp fails on every occluded frame. With a 2-second latent buffer, the same policy typically succeeds in over 90% of trials on the same hardware, though the exact figure depends on gripper speed and occlusion duration. In practice, that buffer also tends to cut the training episodes needed to reach that success rate from roughly 50,000 to around 300, because the policy no longer has to memorize every occlusion pattern through brute repetition. Working memory preserves the mug pose estimate long enough to complete the grasp. Long-term episodic memory preserves the fact that the last handle grasp on that mug slipped, so the next attempt should favor a side grasp.

from dataclasses import dataclass, asdict

@dataclass
class MemoryItem:
    kind: str
    payload: str
    ttl_s: int
    value_for_action: str

    def as_row(self) -> dict[str, object]:
        return asdict(self)

working = MemoryItem(
    kind="working_state",
    payload="mug pose estimate in robot base frame",
    ttl_s=2,
    value_for_action="continue grasp despite one-frame occlusion",
)
episodic = MemoryItem(
    kind="episode",
    payload="failed handle grasp on ceramic mug",
    ttl_s=86400,
    value_for_action="prefer side grasp next attempt",
)
print(working.as_row())
print(episodic.as_row())

{'kind': 'working_state', 'payload': 'mug pose estimate in robot base frame', 'ttl_s': 2, 'value_for_action': 'continue grasp despite one-frame occlusion'}
{'kind': 'episode', 'payload': 'failed handle grasp on ceramic mug', 'ttl_s': 86400, 'value_for_action': 'prefer side grasp next attempt'}

The expected output shows that the two items differ in both lifetime and control function, where ttl_s (time-to-live in seconds: how long the item stays valid before it must be refreshed or discarded) is the field that encodes that lifetime. The first item is about short-horizon continuity. The second is about future adaptation across episodes. Treating them as one generic memory type would hide that design difference.

Practical Recipe

1. Benchmark a no-memory baseline and a short-horizon working-memory baseline.
2. Add only one long-term memory type at a time.
3. Measure whether the added memory improves delayed decisions or repeated tasks.
4. Track retrieval latency, hit rate, and stale-memory usage.
5. Keep all variants on one evaluation panel.