Section 56.2: Spatial, episodic, and semantic memory

"The map knows where, the episode knows when, and the semantic store keeps insisting it knows why."

A Three-Drawer Memory Cabinet

Theory

The three memory families can be written as

$$\mathcal M_{\text{spatial}}=(V,E,\psi), \qquad \mathcal M_{\text{episodic}}=\{(o_{1:T},a_{1:T},r_{1:T},c)\}, \qquad \mathcal M_{\text{semantic}}=\{(k,v,\sigma,\eta)\}.$$

Spatial memory is often metric or topological. Episodic memory preserves trajectories and outcomes. Semantic memory stores abstractions or facts with confidence \(\sigma\) and freshness \(\eta\).

In the spatial tuple \(\mathcal M_{\text{spatial}}=(V,E,\psi)\), \(V\) is the set of places or map cells, \(E\) is the set of edges connecting places that are reachable from one another, and \(\psi\) is a labeling function that attaches geometry (a coordinate, an occupancy value, or a semantic tag) to each vertex or edge. In the episodic tuple, \(o_{1:T}\), \(a_{1:T}\), and \(r_{1:T}\) are the observation, action, and reward sequences recorded over a trajectory of length \(T\), and \(c\) is the outcome or context label attached to that trajectory (for example, success, failure, or a specific fault code). In the semantic tuple, \(k\) is a key (a category or entity), \(v\) is the stored value or embedding, \(\sigma\) is a confidence score, and \(\eta\) is a freshness or staleness measure. These symbols recur throughout the rest of this section: the worked example below assigns concrete numbers to \(\sigma\) and \(\eta\) so the abstract notation maps onto an operational routing decision.

Figure 56.2B makes the routing concrete: a single planning query router inspects each incoming question and dispatches it to whichever of the three stores is built to answer it, and each store runs on its own update cadence.

A memory system that cannot say which kind of question it is answering is not a memory system: it is a search engine waiting to confabulate.

Why episodic memory carries extra weight

Of the three stores the router dispatches to, one carries a weight the others do not, because it is the only one that remembers what physically went wrong. Episodic memory matters in embodied AI because physical actions have irreversible consequences. A robot that has no record of previously jamming a drawer, tripping a circuit breaker, or dropping a fragile object cannot avoid repeating those failures. Spatial memory tells the robot where the drawer is; only episodic memory tells it that opening that drawer at speed caused a wrist-torque fault last Thursday. Without it, every interaction resets to the same prior, and hard-won failure signals vanish after each run.

Mechanically, episodic storage appends a timestamped tuple of observations, actions, and outcomes after each trajectory ends. At query time the retrieval layer searches for episodes whose context embedding is close to the current state. It ranks those episodes by recency and relevance, then returns the top-k records, where top-k means the k highest-scoring matches rather than every record above a threshold. The scale shows what retrieval buys. A robot exploring a 20-room apartment may accumulate 40,000 raw sensor frames per hour. The episodic index condenses those frames into roughly 200 to 400 labeled trajectory records, so the planner reasons over a 100-fold smaller search space without losing the failure signals. The planner then conditions its next action on the retrieved outcome rather than repeating an approach that previously failed.

Keeping episodic recall this sharp depends on not letting it bleed into the other two stores, which is why, in practice, the three live in separate places. Operationally, these memory types usually sit in different toolchains. Spatial memory may live in `tf2` transform trees, occupancy maps, voxel stores, or scene graphs (a scene graph is a node-and-edge structure whose nodes are objects and places and whose edges are spatial relations between them); episodic memory may be written into ROS bags, trajectory databases, or replay buffers; semantic memory may be indexed with FAISS (Facebook AI Similarity Search), pgvector (a PostgreSQL extension that stores embedding vectors and runs nearest-neighbor search inside the database), or a knowledge graph. Separating the stores makes provenance and failure analysis tractable.

These memory types also differ in update cadence. Spatial memory may change every control cycle, episodic memory may be appended after each trajectory, and semantic memory may be updated only after aggregation or human verification. This mismatch is called the memory cadence alignment problem, and conflating those cadences causes either excessive churn or stale abstractions.

Worked Example

A household robot hunting for scissors splits the search across all three stores: spatial for drawer locations, episodic for where it last saw the scissors, and semantic for where households typically keep them.

def validate_memory_queries(payload: dict[str, object]) -> dict[str, object]:
    assert payload, "payload must not be empty"
    return payload

memory_queries = {
    "where_is_drawer_3": "spatial",
    "what_happened_last_time_i_opened_this_drawer": "episodic",
    "where_are_scissors_usually_stored": "semantic",
}
print(validate_memory_queries(memory_queries))

{'where_is_drawer_3': 'spatial', 'what_happened_last_time_i_opened_this_drawer': 'episodic', 'where_are_scissors_usually_stored': 'semantic'}

On a Stretch RE2 mobile manipulator running ROS 2 Humble, each of these queries routes to a different subsystem with a different latency budget. The spatial query hits an OctoMap (a probabilistic 3-D occupancy grid stored as an octree) that a Realsense D435i updates at 10 Hz. A round-trip takes roughly 2 ms. The episodic query replays a compressed ROS 2 bag entry written after the previous grasp attempt. Retrieval from a 72-hour rolling buffer indexed by FAISS typically resolves in 8 to 15 ms. The semantic query calls a pgvector lookup over a 50 k-entry household-object knowledge base, with a median latency of 22 ms. Routing a spatial query through the semantic pathway would add 20 ms of unjustified latency. Worse, it would return a category-level prior ("scissors live in drawers") rather than the current metric pose of drawer 3. The arm trajectory planner would then target the wrong Cartesian goal and trigger a wrist-torque fault at contact (a safety stop triggered when the wrist joint's motor senses more resistance than an expected, unobstructed motion should produce).

A useful evidence artifact here is a memory-routing ledger that records query text, chosen memory type, retrieved record id, freshness field, and downstream action change. That artifact makes it obvious when the robot answered a geometric question with a semantic prior or reused an old episode without checking whether the scene had changed.