Open Source AI Memory Storage - The Deduplication Challenge
The Deduplication Problem in AI Agent Memory
Storing memories for an AI agent is easy. A JSON file, a SQLite database, a vector store - pick your format and append new entries. The hard part is deduplication - knowing when a new memory is really just a rephrasing of something you already stored.
Why Exact Matching Fails
An agent learns "the user prefers dark mode" on Monday. On Wednesday, it encounters "switch to dark theme when possible." On Friday, it records "user likes dark UI." These are the same preference expressed three different ways.
String matching won't catch this. Even fuzzy string matching struggles because the word overlap is low. You need semantic comparison - understanding that these sentences mean the same thing.
Embedding-Based Dedup
The practical approach is to compute embeddings for each memory and check cosine similarity against existing entries before storing a new one. If similarity exceeds a threshold (typically 0.85-0.92), treat it as a duplicate.
from sentence_transformers import SentenceTransformer
model = SentenceTransformer('all-MiniLM-L6-v2')
def is_duplicate(new_memory, existing_memories, threshold=0.88):
new_embedding = model.encode(new_memory)
for mem in existing_memories:
similarity = cosine_similarity(new_embedding, mem.embedding)
if similarity > threshold:
return True, mem
return False, None
The threshold is the tricky part. Too high and you store duplicates. Too low and you merge distinct memories that happen to be related.
The Merge Problem
When you find a duplicate, you can't just discard the new version. It might contain additional details. "User prefers dark mode" and "user prefers dark mode, especially in code editors" overlap but the second adds information.
This means deduplication is really a merge operation - combining the old and new memories into one that captures everything. That requires an LLM call per dedup decision, which adds cost and latency.
What Open Source Solutions Get Right
The best open source AI memory systems treat deduplication as a first-class feature, not an afterthought. They batch dedup operations, use tiered similarity checks (fast embedding check, then LLM verification for borderline cases), and let users review merge decisions.
Memory without deduplication is just a log file with extra steps.
Fazm is an open source macOS AI agent. Open source on GitHub.