Why Desktop AI Agents Skip RAG and Use Structured Markdown for Memory
Why Desktop Agents Skip RAG for Memory
The default architecture for AI agent memory is embed-retrieve - convert information to vectors, store them in a database, query by semantic similarity. It works for general-purpose chatbots. For desktop agents that need fast, reliable recall, it's overkill.
The Hot Path Problem
When an agent is actively controlling your desktop - clicking buttons, reading screens, executing workflows - it needs memory access in milliseconds, not the hundreds of milliseconds that vector database queries take. More importantly, it needs predictable results. Semantic similarity search returns "close enough" matches that can be wrong in subtle ways.
A desktop agent that misremembers your preferred file organization or forgets which app you use for a specific task creates friction every time. The memory system needs to be precise, not probabilistic.
Structured Markdown by Category
The alternative is simple: organized markdown files, each covering a specific category of knowledge.
memory/
preferences.md - User settings and choices
workflows.md - Common task sequences
apps.md - Installed apps and their roles
shortcuts.md - Keyboard shortcuts and aliases
history.md - Recent actions and outcomes
At startup or when switching contexts, the agent loads the relevant category files directly into its context window. No embedding, no retrieval, no similarity threshold tuning. The agent gets exactly the information it needs in a format it can read natively.
When RAG Still Makes Sense
RAG works well for large, growing knowledge bases where you can't predict what will be relevant - think a support agent searching through thousands of documents. Desktop agents have a bounded knowledge domain. Your preferences, your apps, your workflows - this is a finite set that fits comfortably in structured files.
The Maintenance Advantage
Markdown files are human-readable and human-editable. When the agent learns something wrong, you open the file, fix it, and save. Try doing that with a vector database. Debuggability alone makes structured files the better choice for personal desktop agents.
Fazm is an open source macOS AI agent. Open source on GitHub.