Why Standard RAG Is Terrible for AI Agent Long-Term Memory
Why Standard RAG Is Terrible for AI Agent Long-Term Memory
RAG - retrieval-augmented generation - is the default answer when someone asks how to give an AI agent memory. Embed your documents, store them in a vector database, retrieve relevant chunks at query time. It works well for search and Q&A. It falls apart for long-term agent memory.
The Problem with Chunk-Based Retrieval
RAG retrieves text chunks based on semantic similarity. But agent memory is not about finding similar text - it is about understanding relationships, sequences, and context over time.
Consider what an agent needs to remember:
- "Last Tuesday, the user asked me to refactor the auth module, but we rolled it back because it broke the payment flow"
- "The user prefers Zod for validation in TypeScript projects but uses Pydantic in Python"
- "The deploy script needs to run in a specific order: build, test, migrate, then deploy"
These are relational memories. RAG gives you the closest text match, not the connected context. You get fragments without the story.
Graph-Based Memory via MCP
Knowledge graphs store entities and their relationships explicitly. Instead of embedding "the user rolled back the auth refactor," you store nodes (auth module, payment flow, refactor event) and edges (broke, rolled back, caused by).
With MCP - Model Context Protocol - you can expose this graph as a tool the agent queries naturally. The agent asks "what happened last time we touched the auth module?" and gets a structured answer with full context, not a list of similar-sounding text chunks.
When RAG Still Works
RAG is fine for one-shot retrieval - finding documentation, looking up API references, searching through logs. The problem is specifically with persistent, evolving memory that needs to capture cause and effect.
For desktop agents that work with you daily, the memory system needs to understand your workflow history, not just find similar words. Graph-based approaches handle this naturally because relationships are first-class citizens, not inferred from proximity in a vector space.
Fazm is an open source macOS AI agent. Open source on GitHub.