Building Memory Into an AI Desktop Agent

The hardest problem in AI agents is not planning actions or executing them. It is remembering.

Every new session starts from zero. The agent does not know what you worked on yesterday, who your contacts are, how you like your emails formatted, or what project you are currently focused on. You end up re-explaining context that the agent should already know.

The Knowledge Graph Approach

During onboarding, Fazm indexes your files and builds a knowledge graph of entities and relationships:

• Contacts - names, email addresses, how you refer to them ("Sarah" maps to sarah@company.com)
• Projects - what you are working on, which files belong to which project
• Preferences - how you format emails, your writing style, your workflow patterns
• History - what tasks you have completed, what context was used

This graph persists across sessions. When you say "send Sarah the project update," the agent already knows who Sarah is, which project you mean, and what format you prefer.

Recency-Weighted Retrieval

The challenge with persistent memory is knowing which memories matter right now. You might have hundreds of contacts and dozens of projects. Loading all of them into every session wastes context window space.

The approach that works: recency-weighted retrieval. Memories that were accessed recently are weighted higher. Memories that have not been used in weeks decay in priority. The agent loads the most relevant memories for the current task, not everything it has ever learned.

Local Indexing

All of this runs locally. Your knowledge graph is stored on your machine, built from your local files, and never uploaded anywhere. This is critical because the knowledge graph contains some of the most sensitive information about your work - your contacts, your projects, your communication patterns.

---

Fazm builds a local knowledge graph during onboarding for persistent memory across sessions. Open source on GitHub. Discussed in r/AI_Agents and r/ClaudeAI.