Should Robots Remember Maps or Experiences?

1. A Place Is Not a Single State

Before discussing what to put in the map and what to keep in memory, we must first ask what the nature of the world we want to store is. If real space is always in one state, a precise, up-to-date map may be sufficient. However, long-term operation does not make space non-stationary; it reveals that real environments were never a single fixed state to begin with. The Oxford RobotCar collected over 1,000 km of data by repeatedly driving the same route more than 100 times over the course of a year. This data includes not only changes in weather and lighting, but also changes in traffic, pedestrians, construction, and road structure.[3] It is not that the map created from a single survey is wrong, but that it is insufficient to preserve only the correct answer at one point in time.

The value of long-term autonomy does not end with enduring these changes. In four actual deployments, the STRANDS robot traveled 116 km for a total of 104 days, performing custom tasks, and demonstrated a structure that improves system performance using data accumulated during operation.[1] Long-term operation is not a simple durability test, but an opportunity to acquire intelligence specialized for that location.

The way the same place changes is divided into at least four floors.

• Appearance: Observation changes caused by lighting, weather, season, and sensor exposure
• Structure: Accessibility changes caused by doors, furniture, construction, and temporary obstacles
• Routine: Recurring time changes such as crowds, elevators, or cleaning schedules.
• Embodiment: Changes in robot size, loading status, battery and task possibilities create

FreMEn predicts the environmental state at a specific time and improves localization and navigation by modeling occupancy or landmark visibility as a periodic probability instead of a fixed value. [4] The probability that a door is open, the possibility that a hallway is crowded, and the possibility that a landmark is obscured come not only from coordinates but also from the time structure of repeated observations.

The right side of the vertical line in the formula is given condition when interpreting experience. In other words, spatial experience is not a general event that removes place and time, but a conditional record of what was seen and done, and what results were obtained, in what context and embodiment at a specific location and time. A map estimates the state in a given space and time, but experience combines it with action and outcome.

For long-term operation robots, a place is not a single state, but a state distribution according to time, task, and action conditions.

2. What Is the Difference Between a Map and Experience?

If the same location has multiple states, the next question is whether to absorb the differences in a single map update or preserve them as an experience that can be compared and inferred later. SLAM and mapping are the core foundations for robots to organize space into a consistent coordinate system. However, in long-term operation, the map easily converges to a single canonical present, that is, a single state that is considered current. When lighting changes or furniture moves, the map is updated with new observations, and the previous state is easily treated as errors or outdated information.

On the other hand, experience-based navigation does not average the different appearances of the same place into one, but maintains each appearance mode so that each can be recalled. Churchill and Newman performed long-term localization using the visual experiences accumulated by repeatedly driving in different time zones, weather, and lighting for three months. [2] The key is not just to continuously update one map, but to preserve differences so that the past that matches the present can be selected.

• Geometry: empirically observed metric spatial evidence such as free space, surface, shape, distance, and relative position
• Topology: Connectivity, adjacency, reachability and path relationships between places and areas
• Semantics: Identity, category, function of objects, places, people and semantic relationships between them
• Where: spatial anchor connected to the global·local·topological frame to help you find your experience again
• When: Time of observation, duration, repetition period, seasonality and time interval for which the information was valid.
• Observation: Raw evidence directly acquired by the sensor and measurements extracted by the perception module
• Context: Current mission, weather, lighting, crowd, payload, operational status and ambient conditions
• Action: Command, skill, trajectory or interaction executed or attempted by the robot
• Outcome: State change that occurred after the action, success/failure, recovery and observed results
• Uncertainty: How certain each observation/inference/result is and what alternative hypotheses remain.
• Provenance: Source and creation history of what robot/sensor/model and calibration/version it was created with and how it was shared, modified, and verified afterwards.

Therefore, spatial memory is not an alternative that discards maps, but is a higher-level expression that preserves map-like information such as geometry, topology, and semantics along with the spatial basis of each experience. Experience is not additional metadata added on top of a map. It is operational knowledge that changes “you can go” into “if you go this way under these conditions, you are likely to succeed” by linking space to the results of past actions.

3. What Should a Robot Remember About Spatial Experience?

Just distinguishing between map and spatial memory does not determine what should be left and in what units. Even if different states are preserved, they do not function as memory if they cannot be retrieved and used for action. Preserving raw video for a long time does not create long-term memory. A record that cannot be searched or linked to action is just an archive. Conversely, overly compressed semantic labels retain what was present, but lose why a judgment was made and which actions succeeded.

3D Dynamic Scene Graph expresses objects, walls, rooms, places, humans, and robots as hierarchical nodes, and connects containment·adjacency·spatio-temporal relationships with edges.[5] Rather than a compression method that makes the point cloud smaller, it is a starting point for transforming space into a relationship structure that allows questions and planning. Khronos adds the time axis of change to this to construct a dense 4D metric-semantic map that infers short-term movement and long-term change as different processes. [6]

An actionable spatial experience requires the following layers:

Here, the expressions episodic, semantic, and procedural do not claim to replicate human memory as is. It is an engineering abstraction to separate information with different update cycles and search conditions. What is important is not the name, but whether it can express the conflict between current observations and past evidence and even trace back to the consequences of action.

4. Remembering Everything Is Not the Answer to Task Relevance

Defining the necessary memory layers does not mean that all layers must always be saved and loaded at the same resolution. What is important is determined by the current mission, not the space itself. Even in the same building, access permission and elevator waiting time are important for delivery robots, and normal appearance and abnormal changes are important for patrol robots. Cleaning robots should prioritize floor material and contamination frequency, and mobile manipulators should prioritize approach posture and grasp success history. “Memorizing space well” is not a complete specification without specifying a mission.

A memory is task-relevant when it substantially changes the action ranking, uncertainty, or expected costs of the current policy.

Visual Language Maps combines vision-language features into a 3D map, allowing you to query places using natural language landmarks and spatial relationships. [7] ConceptGraphs organizes open-vocabulary objects and relationships into a graph to connect abstract language queries to spatial and semantic inference and planning. [8] This flow is Create a foundation for reading “where is what” in task language.

However, task relevance is not only the final step in selecting search results. Clio bundles task-agnostic 3D primitives under the condition of a natural language task, and preserves the objects and regions required for the task with appropriate semantic granularity according to the Information Bottleneck principle. [9] Since the resolution required for “patrol,” “delivery,” and “person search” is different for the same scene, the task must affect the resolution of the spatial expression itself.

STaR is a recent example of first creating a task-agnostic multimodal memory and extracting a compact, non-redundant task-relevant subset when querying. [12] This direction distinguishes between storage and working memory. In other words, instead of compressing all possibilities into a small working context in advance, only the evidence currently needed is retrieved from widely preserved experience.

5. Semantic Similarity Is Not Decision Relevance

Task relevance sets the goal of what memories are needed, but how to actually find those memories is a separate issue. It is not enough to find a caption similar to the current instruction in the vector database. A record that is semantically close to the sentence “Go to the conference room” may be a conference room taken yesterday on a different floor, and a record that is close in coordinates may be in an old state before construction. Even if the search results are linguistically plausible, there is no guarantee that they are valid for the current action.

The retrieval value of spatial experience must consider at least the following axes:

• spatial proximity and topological reachability
• temporal context, periodicity and freshness
• semantic·task similarity
• Similarity between action and outcome
• Risk, information value and whether current observations are consistent
• Search latency, memory traffic and energy

ReMEmbR stores video caption embeddings along with location and time information in a vector database, and allows the language model to generate repetitive queries to search and infer spatial and temporal memory. [10] This is important early evidence that searchable spatiotemporal memory can aid actual robot navigation. At the same time, it demonstrates the need for a separate runtime policy that verifies retrieved memories with current observations and determines what to load within a limited context budget.

The retrieval just before the action must operate in a closed loop as follows:

1. 1 Question The planner creates a question that can change the current decision.
2. 2 Retrieve Find candidate episodes by space, time, object, and action index.
3. 3 Rerank task Rerank by relevance, freshness, risk and variety.
4. 4 Verify Verify the validity of memory through live perception or active sensing.
5. 5 Act planner·verifier reflects verified evidence in actions and fallback.

The moment spatial experience becomes intelligent is not when it is recorded, but when it changes the ranking and uncertainty of current actions.

6. The Ability to Remember Includes the Ability to Forget Well

Even if you design a decision-relevant retrieval, if the input memory increases without limit, the retrieval quality and execution cost deteriorate again. The problem of retrieval ultimately leads to the question of what to keep and what to integrate or discard. The robot's sensors don't stop. Leaving every moment of video, point cloud, caption, location, and action not only increases storage capacity. Similar episodes dominate search results, older experiences score higher than current observations, and memory traffic and retrieval latency continue to increase.

Therefore, forgetting is not a post-scavenging process that frees up storage space, but is part of the memory architecture. However, it should not be as simple as “delete when it gets old.” Repeated observations should be consolidated into temporal patterns, duplicate episodes should be merged, and facts that change frequently should be given a short validity period. Conflicting beliefs should not be overwritten immediately, but should be managed as a version by adding valid_from, valid_to, confidence, and source.

There is no need to delete it right away just because the freshness is low. If old memories are important for current decisions, the robot can revisit them through active perception. A good forgetting policy is not a binary choice between memory and deletion, but rather a choice between preservation/compression/attenuation/reobservation, whichever is most economical.

7. Spatial Experience Must Preserve Evidence of Actions and Outcomes

Selective forgetting requires criteria to determine what information can change behavior. In that standard, what is more important than the scarcity of observations is what actions led to what results. It is difficult to determine the next action based solely on the observation that “the west gate was closed at 7 o’clock at night.” It becomes affordance and recovery knowledge only when the record also captures whether the robot pulled the handle, used an access card, carried a payload, and succeeded by detouring through the north entrance.

SPTM created a topological memory with the observed location as a node and the movement possibility as an edge, and searched for relevant nodes in the current observation and used them for goal navigation. [13] This shows the principle of turning the trajectory experience into a planning substrate instead of rewatching all past images. However, since it is a setting that solves the maze for the first time after a short search, it is not in itself evidence of lifelong memory.

Actions must rewrite memories, not just read them. DynaMem adds new objects to the spatio-semantic memory according to RGB-D observations and removes moving or missing evidence online. [11] If the results of navigation or manipulation are different from expectations, the robot must not only modify the plan, but also leave a note under what conditions the existing belief was incorrect.

Not all navigation decisions require a complete causal model. Path costs can be lowered simply by predicting repeating patterns. However, when selecting an action that changes the environment, such as opening a door or moving an object, and generalizing it to new conditions, action-conditioned outcome becomes more important than simple correlation.

8. Experience 2.0: A Closed-Loop System, Not an Archive

So far, we have looked at the requirements for representation, retrieval, forgetting, and action-outcome evidence, respectively. The remaining question is in what order these elements should be connected and update each other during operation. If you combine this into one structure, long-term spatial memory should not be a single database, but a closed loop on different time scales. The structure below is not the name of an already established standard, but a design proposal for linking long-term spatial experience with actual behavior.

The relationship between the original and the summary is also important. Leaving only the caption or embedding makes storage and retrieval easier, but the discarded visual information cannot be restored later when a completely different mission appears. All raw streams cannot be permanently stored onboard. A realistic structure is a hierarchical memory that leaves low-cost traces and provenance, creates indexes and summaries for each task at multiple resolutions, and allows return to the original evidence or fleet-level archive when necessary.

9. Spatial Experience Must Not Be Trapped Inside a Single Robot

Even if closed-loop memory works well within a robot's lifetime, times and conditions that the robot has not seen are still left blank. If there are multiple robots operating in the same space, the next unit of expansion is not individual memory but the collective experience of the fleet. The experience of an individual robot is necessarily partial. One robot sees morning lights, another experiences lunch crowds, and yet another fails on a slippery floor after rain. If each robot only maintains its own memory, the fleet relearns the same location multiple times and pays the same failure cost over and over again. Conversely, if observations and action results can be shared, the limited lifetime of one robot can be extended to the cumulative operating time of the entire fleet .

multi-robot SLAM already shows the value of spatial information sharing. Kimera-Multi aligned the trajectories and local meshes of multiple robots using only peer-to-peer communication and distributedly constructed a globally consistent metric-semantic 3D map that is robust to outliers. [15] However, combining geometry and semantic labels is only a starting point. What the robot actually attempted, under what conditions it succeeded or failed, and whether that experience is still valid cannot be conveyed through map fusion alone.

An early example that directly addressed this issue is Checkout My Map. This study proposed a version-control structure in which multiple vehicles share, update, and reuse each other's experience maps for long-term visual localization, and only the currently relevant subset, rather than the entire archive, was distributed to resource-constrained robots. [14] In other words, in fleet memory, the key is not to replicate all data, but to manage who created what experience and when and which robot needs it now. It is .

Here, shared experience is not a separate expression equivalent to shared map, but a higher-level schema that can include it as an optional space layer. If a common metric-semantic map is constructed, episodes can be matched to that map, but there is no requirement that there be a completed shared map. Robots can share who attempted what, when, in what embodiment, and what results were obtained, using only place identity or local/topological references.

Distinguish between shareable facts and embodiment-dependent experiences

Facts observed by different robots are not transferred to the same value. Although it is useful for a drone to indicate the location of a staircase, the fact that the drone has traversed that path is not evidence of the traversability of a ground robot. The width of the door that a small delivery robot passes through, the handle that a humanoid opens, and the energy cost of a robot carrying a heavy load cannot be directly applied to other embodiments. Therefore, shared experience must separate facts and affordability, and attach robot body, sensor, controller, payload, and skill version to the latter as conditions.

Collective memory requires a sharing policy

There is already evidence that fleet-level data can improve the performance of other robots. OpenBot-Fleet built a closed loop that learns navigation data collected by 72 robots in the cloud and distributes them back to the fleet, reporting a success rate of over 80% in unseen home. [16] Open It was visible.[17]

The failure of one robot should not be an event that another robot must experience again, but should be a prior for the next robot.

10. Evaluate Behavioral Improvement, Not the Amount Remembered

The claim that a structure that preserves experience longer and shares it across multiple robots is useful goes beyond research hypotheses only when testing whether actual behavior improves. Therefore, evaluation should not stop at storage capacity or recall ability. Memory QA accuracy is a necessary diagnosis, but not a sufficient evaluation. Even if a robot correctly answers where a cup was yesterday, it fails as an embodied system if it cannot disregard that stale memory after the cup has moved today. Conversely, even if you cannot recall all the details, it may be a more useful memory if you avoid dangerous passages and recover quickly after failure.

The benchmark should also include repeated operations rather than one fixed episode. While operating the same space over several weeks, controlled changes such as door locks, object movement, construction, and crowds at different times must be included, and different states of the same appearance and the same state of different appearances must be configured together. Compare no-memory, static map, full-history oracle, and task-relevant memory, but measure not only QA but also actual navigation/manipulation results.

11. Spatial Experience Is a Computational Resource for Runtime Intelligence

If we shift the focus of evaluation from memory itself to behavioral change, the final role of spatial experience becomes clear. Memory is not an output that provides a complete answer, but is a resource of runtime computation that is selectively called in the current situation. Even if task-relevant memory is prepared, it is not automatically determined what to search for at any moment, which side to trust if it conflicts with the current observation, whether to spend time on additional observations, or whether to reuse past recovery. Memory is the input for reasoning and planning, and reasoning is the runtime policy that calls and verifies memory.

Reasoning without memory requires discovering the world again every time, and memory without reasoning blindly repeats past actions.

The first guiding research question asks what robots should take from the past. The second question asks how runtime computations should be distributed among perception, reasoning, memory, planning, and verification, based on the experience and current observations thus retrieved. The two questions are not separate. More memory creates more reasoning cost, better retrieval reduces the runtime compute required, and uncertainty determines when to invoke the slow path.