Section 32.5: Multimodal memory

"Without memory, every observation is a first meeting. The agent that forgets what it saw ten seconds ago cannot be trusted with a task that takes eleven."

A Persistent Collaborator

A robot that perfectly recognizes your red mug will still march confidently to an empty counter if the only thing it remembers is that the mug looked good from there ninety seconds ago, before someone carried it to the sink. That gap between recognizing an object and trusting where it now is, is exactly what multimodal memory must close, by storing observations as time-stamped, frame-anchored state hypotheses and designing retrieval that weights freshness and uncertainty, not just embedding similarity. Figure 32.5A shows the target architecture: episode frames and text annotations land in a key-value store (a lookup table where each stored item, the value, is retrieved by a distinct identifier or embedding, the key), and a cross-modal retrieval module (a search step that can match a query in one modality, such as text, against stored items in another, such as images) recalls relevant past observations when a matching scene or instruction recurs. Read the figure as a memory-indexing contract. Multimodal memory helps only when the retrieval key, stored evidence, temporal validity, and action consumer are explicit enough to prevent stale observations from steering the robot.

What Multimodal Memory Must Remember

A useful robot memory is not just a collection of captions or embeddings. It must bind semantics to geometry and time, a property sometimes called spatiotemporal grounding of observations, and without it retrieval cannot support planning. A memory item might say: "mug_2, red, left of sink, pose in frame map, confidence 0.73, observed at 14.2 s, last revalidated at 15.1 s." Without pose and time, the memory cannot participate in planning. Without semantics, it cannot answer instruction-conditioned queries.

By the end of this section you should be able to do three concrete things: design a memory schema with the fields listed in the table below, write a freshness- and uncertainty-aware retrieval score like the one introduced next, and diagnose a stale-memory failure by naming which missing field (timestamp, frame id, or invalidation rule) let it happen. The worked examples and the lab later in the section give you practice at each of these in turn.

This makes multimodal memory a close relative of the world-state machinery in SLAM and map uncertainty. The difference is that Chapter 32 adds language labels and task-conditioned retrieval on top of geometric state.

Retrieval Score With Freshness And Uncertainty

Deciding that an entry must be able to change behavior is only half the design; the other half is a scoring rule that decides which entry earns that influence at retrieval time.

Suppose each memory item \(m_i\) contains a visual embedding \(v_i\), a text summary embedding \(t_i\), an age \(\Delta t_i\), and an uncertainty penalty \(u_i\). A practical retrieval score is

The first two terms reward semantic and visual match. The last two terms penalize staleness and uncertainty. Many toy memory demos omit this part, but robotics needs it because an old perfect match can be less useful than a recent approximate match. Consider the object-query workload on a Stretch RE1 or a Fetch mobile manipulator running CLIP embeddings over a Habitat 2.0 ReplicaCAD apartment (a photorealistic simulated apartment scene used as a standard benchmark for indoor navigation and manipulation research). One failure dominates it. In a 50-object scene, the same mug or sponge is observed from multiple viewpoints, which leaves several near-duplicate entries in the store, and the visually strongest one is frequently the oldest. A retrieval rule that ranks on embedding cosine alone sends the base to the last verified pose even after a household resident moves the object. The dominant error is therefore confident navigation to an empty location, not a recognition miss. The \(-\gamma\,\Delta t_i\) and \(-\delta\,u_i\) terms change the ranking so a recent, frame-consistent observation outranks a stale near-twin. On a navigation budget, the base then spends its meters reaching objects that are actually there instead of recovering from retrievals that point at stale state.

Worked Example

Code Fragment 1 makes that ranking rule explicit by combining visual match, text match, age, and uncertainty. This is the minimum interface needed before a vector database becomes genuinely useful for robotics.

# Rank memory entries by semantics, visuals, freshness, and uncertainty.
# The best entry should be useful now, not only historically descriptive.
# Age and uncertainty penalties keep stale evidence from dominating retrieval.
entries = [
    {"id": "mug_2", "visual": 0.82, "text": 0.77, "age": 0.4, "uncertainty": 0.08},
    {"id": "mug_2_old", "visual": 0.91, "text": 0.85, "age": 8.6, "uncertainty": 0.11},
]

def memory_score(item, alpha=0.45, beta=0.35, gamma=0.03, delta=0.40):
    return alpha * item["visual"] + beta * item["text"] - gamma * item["age"] - delta * item["uncertainty"]

ranked = sorted(((item["id"], round(memory_score(item), 3)) for item in entries), key=lambda pair: pair[1], reverse=True)
print(ranked)

The expected output is a ranked memory list where the fresher entry, mug_2, outranks the older but visually stronger one. That ordering is what the reader should check first: if mug_2_old still won, the scoring policy would be under-penalizing age and the memory system would behave more like static retrieval than embodied state estimation.

With this rule, a recent but slightly weaker observation can outrank an older perfect match. That behavior is often exactly what the planner needs when deciding whether to act immediately or revisit a location. A memory that ignores time is not a state estimator; it is a museum exhibit: accurate about the past, useless for the present.

Code Fragment 2 shows the maintained pattern with a metadata-aware vector search.

# Query a vector memory with metadata filters for recency and source frame.
# pip install lancedb pyarrow
query_embedding = [0.12, -0.33, 0.48, 0.27]
filters = {"age_seconds_lt": 2.0, "frame_id": "map"}
mock_results = [
    {"object_id": "mug_2", "confidence": 0.74, "age_seconds": 1.1, "frame_id": "map"},
    {"object_id": "plate_1", "confidence": 0.63, "age_seconds": 3.4, "frame_id": "map"},
]
filtered = [row for row in mock_results if row["age_seconds"] < filters["age_seconds_lt"] and row["frame_id"] == filters["frame_id"]]
assert filtered
print({"top_object": filtered[0]["object_id"], "confidence": round(filtered[0]["confidence"], 2), "query_dim": len(query_embedding)})

The expected output is a single top retrieval whose object id and confidence survive both the vector similarity step and the metadata filter on age and frame. In other words, the system is not only finding a semantically similar memory, it is finding one that is still recent enough and frame-consistent enough to guide a new action.

Memory Schema For Embodied Systems

The retrieval score and its metadata filters are only as good as the fields each entry actually stores, so the next question is what a durable memory record must contain.

A durable schema usually needs at least these fields: object or region id, visual embedding, text summary, pose or frame reference, timestamp, confidence, source sensor, and invalidation rule. The invalidation rule is often overlooked, but it matters whenever a door can open, a person can move an object, or the robot itself can change viewpoint enough to break correspondence.

The most valuable memory audit is to replay one failure episode and ask which memory entry should have been ignored, updated, or invalidated. That question is usually more informative than asking which retrieval model had the best standalone benchmark score. In embodied systems, the policy around memory often matters as much as the embedding model inside it.

The invalidation rule matters because the physical world changes without warning: a person moves an object, a door swings closed, an arm occludes a surface. A memory that never expires directs the robot to locations where objects no longer exist, producing wasted motion, arm collisions, and failed grasps that a static web index would never provoke.

Mechanically, an invalidation rule is a predicate evaluated at retrieval time. Common forms include time-to-live (invalidate if age exceeds a threshold), mobility class (invalidate movable objects sooner than fixed furniture), and event-triggered flags (invalidate when a door-open or gripper-contact sensor fires). When the predicate returns true the entry is suppressed from the ranked output and replaced with a reobservation request, so the planner acts on current evidence rather than stale state.

A Failure Without an Invalidation Rule

Consider a specific case. A home robot stores "keys on the kitchen counter" with a visual embedding and text summary but no pose frame and no invalidation rule. Thirty minutes later the user asks "where are my keys?" The robot retrieves the entry (embedding match: 0.88) and navigates to the counter, only to find the keys gone, because a family member moved them. Without a timestamp, the system cannot decide whether to trust the entry or reobserve. Without an invalidation rule tied to object mobility, it repeats the same confident mistake on every query. A timestamp age of 1800 s and a mobility flag of "movable" would have triggered a reobservation policy instead of a direct navigation command, turning a failed retrieval into a correct "I should check before committing."