From 800 Redundant Lines to 30 Curated Pointers - Memory Deduplication in AI Agents
Memory Deduplication in AI Agents
After running an AI agent for a few weeks, its memory file had grown to 800 lines. Most of them said roughly the same thing in slightly different ways. The agent kept appending new observations without checking if it already knew something similar. The result was a bloated memory that wasted tokens and added noise.
The fix was simple: switch from INSERT to UPSERT. Instead of always appending, check if a similar memory exists first. If it does, update it. If it does not, add it.
Why Memory Bloat Happens
Agents default to appending because it is safe. Every observation gets recorded. Nothing gets lost. But this creates three problems:
- Token waste - the agent loads 800 lines of memory when 30 would carry the same information
- Contradictions - old observations conflict with new ones, and the agent does not know which to trust
- Retrieval noise - when searching memory, redundant entries dilute the relevance of search results
The UPSERT Pattern
Before writing a new memory, check existing memories for semantic overlap:
- If a match exists with the same meaning, skip the write
- If a match exists but the new observation adds detail, update the existing entry
- If no match exists, create a new entry with a clear, specific statement
This is not just deduplication - it is curation. Each memory entry becomes a precise pointer to a fact, preference, or pattern instead of a raw log entry.
Results in Practice
After applying this pattern, the memory file went from 800 lines to 30 curated entries. The agent loaded faster, made better decisions, and stopped contradicting itself. Each memory entry carried genuine signal.
The Broader Lesson
Memory is a design decision, not a storage problem. Treating it like an append-only log creates the same mess as never deleting emails. Treating it like a curated index keeps your agent sharp and context-efficient.
Fazm is an open source macOS AI agent. Open source on GitHub.