The Big Gap in Desktop Agents - They Forget Everything Between Sessions
Every Other App Remembers. Your AI Doesn't.
Your browser remembers your bookmarks, passwords, and frequently visited sites. Your email client remembers contacts and sorts mail into categories you have trained. Your IDE remembers your settings, snippets, and recent projects. Notes apps sync across devices. Calendars remember every event since you signed up.
Your AI agent? It wakes up with amnesia every single session. You explain the same context, re-describe the same preferences, and re-teach the same workflows. Every time.
Why This Is the Biggest Gap
Session memory is not a nice-to-have feature. It is the difference between an assistant and a tool. Tools do not learn. Assistants do. When your AI agent forgets everything between sessions, it can never become more than a sophisticated tool.
Think about what a human assistant learns in their first month on the job. Who you prefer to CC on emails. How you like meeting notes formatted. Which Slack messages are urgent and which can wait. Which vendor contacts are reliable and which overpromise. An AI agent that forgets every session can never build this kind of understanding.
The core problem is architectural, not a feature gap. Large language models are stateless by design. Each API call starts with an empty context window. Even models supporting 200K token windows only maintain context within an active session - the moment you close the window, it is gone. The statelessness is not a bug in those systems; it is fundamental to how they work at scale.
Why Bolted-On Memory Falls Short
Some tools bolt on memory features - typically a list of facts the model was told to remember. That is closer to a sticky note than real memory.
The limitation shows up quickly in practice. A sticky-note memory knows "user prefers concise responses." It does not know that last Tuesday you asked for a detailed breakdown on the infrastructure cost question, and that you only want brevity for status updates, not technical analysis. It cannot distinguish between contexts because it has no structure - just a flat list of assertions.
Real memory requires understanding relationships. Knowing that Sarah is your design lead, that she prefers Figma over Sketch, that she usually responds to emails within an hour, and that last Tuesday she mentioned a deadline change. These connections form a knowledge graph that makes the agent genuinely useful over time.
The Technical Architecture That Works
The research community has converged on two approaches that actually work in production:
Temporal knowledge graphs (Zep's architecture) store memory as graph nodes with timestamps, tracking how facts change over time. If your preference for meeting note format evolves, the graph retains the history rather than overwriting. Multi-hop queries work - "what did I discuss with Sarah about the Q2 roadmap?" retrieves through person -> project -> conversation edges.
Semantic memory stores (Mem0's approach) extract discrete facts from conversations and store them as structured nodes with embeddings. New information is reconciled against existing facts before storage, automatically resolving conflicts. The Mem0 team reports a 91% reduction in response time compared to loading full conversation history, because retrieval fetches only what is relevant rather than dumping everything.
Both approaches support sub-200ms retrieval even as memory grows into thousands of facts.
What Persistent Memory Looks Like in Practice
A well-designed persistent memory layer does five things:
1. Extracts facts automatically - after each session, the agent identifies what it learned and structures it. Not dumping the conversation log - extracting the signal.
2. Reconciles against existing knowledge - if the agent knew "Sarah manages design" and learns "Sarah moved to product last month," it updates rather than duplicates.
3. Scopes retrieval to current context - loading memory for a coding session should surface dev preferences, not your travel booking habits from last month.
4. Supports on-demand deep retrieval - the active context window should be small and relevant, but older memories should be accessible when explicitly needed.
5. Decays stale facts - a preference about a project you wrapped up eight months ago should not compete for context with your current work.
A Minimal Implementation
Here is a starting point for session memory that persists between runs:
import json
from pathlib import Path
from datetime import datetime
class SessionMemory:
def __init__(self, memory_path: str = "~/.agent_memory.json"):
self.path = Path(memory_path).expanduser()
self.data = self._load()
def remember(self, key: str, value: str, context: str = "general"):
"""Store a fact with timestamp and context tag."""
self.data["facts"][key] = {
"value": value,
"context": context,
"updated": datetime.now().isoformat(),
"recall_count": self.data["facts"].get(key, {}).get("recall_count", 0)
}
self._save()
def recall(self, context: str = None) -> dict:
"""Retrieve facts, optionally filtered by context."""
facts = self.data["facts"]
if context:
facts = {k: v for k, v in facts.items() if v["context"] == context}
# Update recall counts
for key in facts:
self.data["facts"][key]["recall_count"] += 1
self._save()
return facts
def _load(self) -> dict:
if self.path.exists():
return json.loads(self.path.read_text())
return {"facts": {}, "sessions": []}
def _save(self):
self.path.write_text(json.dumps(self.data, indent=2))
This is flat-file storage, not a graph database - but it demonstrates the essential pattern: facts persist between process restarts, context scopes retrieval, and recall counts feed into triage decisions.
Why Local Memory Changes the Dynamic
A local knowledge graph stored on your machine changes the dynamic completely. Every interaction adds to the agent's understanding of how you work. After a week, it knows your common workflows. After a month, it anticipates what you need.
This only works locally because the data is deeply personal. Your contact patterns, work habits, and communication style are exactly the kind of information you do not want sitting on someone else's server. The AI tools that accumulate personal context on their own infrastructure face a trust problem that local agents sidestep entirely.
The gap between "AI tool you use" and "AI agent that works for you" is a memory problem. Solving it - with structured local knowledge graphs, decay policies, and scoped retrieval - is what turns a demo into infrastructure.
Fazm is an open source macOS AI agent. Open source on GitHub.