待翻譯：Building Shared Memory for AI Agents in Notion

AI 服務暫時不可用，以下為來源正文，待恢復後補全翻譯。

←All posts Building Shared Memory for AI Agents in Notion As software engineering changes, tribal knowledge is increasingly concentrated not only in a single person's mind, but in a single agent session. Here at Notion, we kept having to manually extract experiential learnings as skills or documentation and share them with the team in-repo. But that doesn’t scale. Even as agents got really good at inferring or discovering information in source, this experiential knowledge—which is separate from user preferences—remained a tangible gap. The inverse is also true: agents would often discover information that is tangential to their current workstream but helpful, and those tasks, refactors, and todos would later get lost (especially if it’s an unattended session!). We came to realize that every org has its own lore and must be facing similar problems. We decided it was time to fix that with Notion. Give Your Agents a Memory Lore is the tool we built for that gap: shared, persistent memory for agents, backed by Notion. It is open source under the MIT license at github.com/makenotion/lore. We wanted to make agent experiences durable concepts, stored in Notion, and accessible by both humans and agents. There were multiple items we wanted to ensure were not lost: Experiential knowledge. This is the kind of information that cannot be inferred by a future agent and must be re-experienced. The lengthy rediscovery phases can be short-circuited early or prevented entirely. Follow-up tasks. Often, when an agent is working on a specific task, the agent makes note of specific, unrelated, tangential information. Due to the nature of an agent session, this information is ephemeral by nature: if the human operator was not actively reading the transcript, these un-filed followup items are gone forever. Decisions records. Some teams have a healthy practice of keeping ADRs. Most teams strive for this, but aren’t able to do so in real-time as decisions evolve. Procedures. These are specific workarounds or recipes. Things that could be encoded into a skill but may have usage too infrequent to warrant it. These items become Notion pages that a later session can read back, or that a human can reference as needed. Any MCP-compatible agent harness can reference the vault, and as a result agent sessions are empowered by agents past. Native memory systems are hyper-localized. In-harness systems are often tied to a given harness or given model provider. File-backed systems are often localized to a single machine. We really wanted to experiment with the concept that the agents on one person’s desk could learn from the experiences of another person’s agents—perhaps even from another part of the organization. We wanted a shared platform for the things that should be carried forward. How it works A Lore vault is a Notion page with five databases: Database What it stores Projects The project, person, team, or agent scope Topics The subject areas inside that scope Memories Narrative context, notes, procedures, tasks Entities Named things the memory graph can refer to Facts Structured subject-predicate-object assertions They have relations to one another, across these tables, to help in surfacing information and to help in fuzzy searching: for example, facts can have relations to topics or memories. This is an opinionated structuring of data, but structured data allowed for agents to have a predictable search structure. Agents access Lore through the MCP, with various lore-* MCP tools. Humans, however, get access to that same information either through Notion directly (after all, these are all Notion pages!), or via a Lore CLI. The innate design is driven largely through hooks, which are intended to make the standard path entirely hands-off: agent sessions start with relevant context, are prompted to store or fetch information as needed, do their work, and quietly save new information in the background. We weren’t interested in single-player solutions. Notion felt like the natural home for a team-focused, multiplayer service. The team already stores operational information in Notion, plans together, builds together, and more. Notion’s built-in version control and version history mechanisms combined with its familiarity are added bonuses. We wanted the information vault to be human-centric, but agent powered. Memory has to stay clean The first bad version of a memory system is a junk drawer with an API. All of the transcripts, all of the text, and all of the sessions in a single folder don’t do much if: Stale information can otherwise harm an agent that would have otherwise been successful. Irrelevant information is surfaced to those agents. The right information is impossible to find. By default, facts expire. Information must be continually reinforced or it will degrade into irrelevance. Addendums to information are handled with known relational values (supersedes, scoped, conflicts_with, etc) that make the relationship to the previous entry clear. The CLI provides tooling for finding orphaned facts, duplicated information, and other memory debt. Memory is something that must be periodically maintained, either manually or through use of a specialized agent. Benchmark results seemed to make it clear that memory largely helps only when it’s available, specific, and retrieved at the right time – but can also harm if it’s vague, redundant, stale, or has no bearing to the task at hand. This is even more pronounced on smaller models. What we measured We wanted some quantifiable numeric values to help guide us through the process. Something feeling like it works while it silently harms is not uncommon. We sought to answer with two relevant benchmarks: 1. Retrieval. Are we pulling the right information? 2. Model-hard eval. These are things that were just outside of model capability, that models could not reliably succeed in doing. This was to answer the question: When the memory is available, does it help? Retrieval We used the open SkillRet data-set, which has 4,997 queries, 6,660 skills, and 8,347 relevance judgments. We import the skills into a Lore eval vault as procedure memories, then give a Codex agent read-only Lore instructions plus search and expand tools. To pass, it has to use Lore, surface the expected memory, expand it, select the right SkillRet target, and apply that target in the answer. We selected 500 queries at random, and found the following: Metric Result Queries 500 Passed 420 Failed 80 Success 84.00% 95% Wilson interval for success 80.53-86.95% Lore tool use 100.00% Target surfaced in tool results 91.80% Target expanded 91.60% Target selected / answer applied 84.00% Write attempts 0 Recall@10 0.4157 This allowed us to scope whether or not we’d be able to identify the right thing to begin with – it provides a ceiling for the maximum number of situations that Lore would be able to assist in. If we are unable to surface the right data, then the memory can’t help. If we surface bad data, then the tool could hurt. Model evaluation The second, and harder question to answer: When the right memory is available, does it improve actual agent work? The design of this process was to pull several large OSS repos pinned at specific SHAs. Their histories erased by condensing them into a single commit. We devised a number of feature requests, bug fixes, or performance improvements that the model could not reliably deliver without Lore. Then, we measured how often Lore lifted a model performance, as well as how often it harmed model performance. Comparison Paired scenarios No-memory passes Memory passes Lift / harm McNemar p Contested recovery seeded-lore vs. no-memory 149 82 102 27 / 7 0.0008 45.76% What this tells us is: when we have relevant memories in the vault, we are able to improve model performance on net. When we filter the data set only to the hardest tasks (those that the no-memory agent never succeeded in doing), the seeded memory agent recovered about 46% of failures! A key takeaway is not that memory magically fixes agents. It means that available, relevant memory changes outcomes—and that ultimately stems from a vault of information that is well kept and managed. One of our internal teams piloted Lore for some time, and built up a large vault. With minimal upkeep, it was shaped such that: 55%-60% of conversations / note memories were experiential and worth keeping. 15%-20% were likely duplicates or near-duplicates. 8-12% were largely information that the agent can infer on its own. 3-5% were session notes. and ~13% had no valuable information in them whatsoever. We did not benchmark against this vault. It stands to reason that we would see degraded performance in messier vaults, but the conclusions remain the same: When the right memory is available, agents can use it, and we see real lift. Memory quality matters. Optimizing for memory quality on a system that has potential to improve agent performance is a much more interesting problem than the one we started with. Try it Lore is MIT-licensed on GitHub: github.com/makenotion/lore. It was built as a 20% project, largely as a pointed exploration of these systems into an off-label use of Notion. Add it as a dev dependency, create a Notion Personal Access Token, point a .lore.yaml at a vault page, and wire it into your assistant. If you build on Lore, fork it, or take it somewhere we did not expect, we want to see it. Issues and pull requests are open at github.com/makenotion/lore/issues. Share this post Get pricing help, demos, use-cases, and more. Request a demo