Context Windows Are Not Memory

Matthew Diakonov··2 min read

Context Windows Are Not Memory

The most common misconception about LLMs is that the context window is the model's memory. It is not. The context window is working memory - the scratchpad where current processing happens. Memory is what persists after the session ends.

The Distinction Matters

Working memory (context window):

  • Exists only during the current conversation
  • Has a fixed size limit
  • Holds everything the model is currently "thinking about"
  • Gone when the session ends

Memory (persistent storage):

  • Survives across sessions
  • Can grow indefinitely
  • Requires explicit read and write operations
  • Needs retrieval mechanisms to find relevant information

When people complain that AI "forgot" what they told it yesterday, they are expecting memory from a system that only has working memory.

Why Agents Need Both

A useful AI agent needs:

  • Working memory (context window) for the current task - the file it is editing, the conversation it is having, the plan it is executing
  • Short-term memory for the current session - decisions made earlier, errors encountered, user preferences expressed
  • Long-term memory for cross-session continuity - project context, learned preferences, past interactions

Most agent frameworks only provide the first one. Some bolt on the second through conversation history. Almost none handle the third well.

Building Real Memory

Practical approaches to agent memory:

  • File-based memory - write important context to files that persist between sessions (this is what CLAUDE.md does)
  • Database-backed memory - store structured information about past interactions, preferences, and decisions
  • Embedding-based retrieval - index past conversations and retrieve relevant snippets when similar topics come up
  • Summary chains - compress long interactions into summaries that can be loaded into future sessions

The key insight is that memory is an engineering problem, not a model capability problem. You build it around the model, not into it.

Fazm is an open source macOS AI agent. Open source on GitHub.

More on This Topic

Related Posts