Context Windows Are Not Memory
Context Windows Are Not Memory
The most common misconception about LLMs is that the context window is the model's memory. It is not. The context window is working memory - the scratchpad where current processing happens. Memory is what persists after the session ends.
The Distinction Matters
Working memory (context window):
- Exists only during the current conversation
- Has a fixed size limit
- Holds everything the model is currently "thinking about"
- Gone when the session ends
Memory (persistent storage):
- Survives across sessions
- Can grow indefinitely
- Requires explicit read and write operations
- Needs retrieval mechanisms to find relevant information
When people complain that AI "forgot" what they told it yesterday, they are expecting memory from a system that only has working memory.
Why Agents Need Both
A useful AI agent needs:
- Working memory (context window) for the current task - the file it is editing, the conversation it is having, the plan it is executing
- Short-term memory for the current session - decisions made earlier, errors encountered, user preferences expressed
- Long-term memory for cross-session continuity - project context, learned preferences, past interactions
Most agent frameworks only provide the first one. Some bolt on the second through conversation history. Almost none handle the third well.
Building Real Memory
Practical approaches to agent memory:
- File-based memory - write important context to files that persist between sessions (this is what CLAUDE.md does)
- Database-backed memory - store structured information about past interactions, preferences, and decisions
- Embedding-based retrieval - index past conversations and retrieve relevant snippets when similar topics come up
- Summary chains - compress long interactions into summaries that can be loaded into future sessions
The key insight is that memory is an engineering problem, not a model capability problem. You build it around the model, not into it.
Fazm is an open source macOS AI agent. Open source on GitHub.