Building a Desktop Agent in Go with Neo4j Memory - Why the Architecture Choices Matter
Building a Desktop Agent in Go with Neo4j Memory - Why the Architecture Choices Matter
When the dominant open-source desktop agent does not fit your mental model, you build your own. OpenLobster is what happens when someone chooses Go over Python and a graph database over flat files for agent memory. Those choices are not arbitrary - they reflect specific technical constraints that Python and flat-file storage handle poorly.
Why Go for a Desktop Agent
Most AI agent projects default to Python because the ML ecosystem lives there. NumPy, PyTorch, Transformers, LangChain - all Python. But a desktop agent is not a training pipeline. It is a long-running application that manages multiple concurrent operations simultaneously:
- Watching the screen for state changes
- Processing user input events
- Executing multi-step actions
- Updating and querying the memory layer
- Managing background sync operations
Python's Global Interpreter Lock (GIL) prevents true parallel execution across threads. A Python desktop agent serializes these operations through a single interpreter lock, which creates latency in exactly the situations where responsiveness matters most - when the user is waiting for an action to complete.
Go handles concurrency natively. A goroutine costs roughly 2KB of memory at startup (compared to megabytes for OS threads) and scales to thousands of concurrent operations. The language makes concurrency idiomatic - adding go before a function call is all it takes to run it asynchronously. Channels provide safe communication between components without manual synchronization.
The performance difference is measurable. Benchmarks comparing equivalent Python and Go concurrent programs show Go is typically 40x faster in I/O-bound scenarios and handles 10x more concurrent goroutines before performance degrades. For a desktop agent that is constantly watching, listening, and acting, that ceiling matters.
The binary deployment advantage is also real. Go compiles to a single static binary. A user installs the agent by downloading one file. No Python version management, no virtual environments, no dependency conflicts. This aligns with how a desktop application should work.
Why Neo4j for Agent Memory
Flat-file memory - markdown files, JSON blobs, SQLite tables - works for simple recall. "What did the user tell me about their preferences?" answered by reading a file. But agent memory has a relationship structure that flat files cannot express efficiently.
A contact is connected to projects. Projects connect to tools and workflows. Workflows connect to recurring tasks. Those tasks connect to the outcomes the agent has observed. These relationships matter when the agent needs to reason about context: "What tools does this person's project use?" requires joining four tables in a relational schema, or traversing four hops in a graph.
Neo4j stores relationships as first-class data. A graph traversal that would require four SQL joins executes as a single Cypher query:
MATCH (contact:Person {name: $name})-[:WORKS_ON]->(project:Project)
-[:USES]->(tool:Tool)
RETURN tool.name, tool.last_used
ORDER BY tool.last_used DESC
This is not just a syntactic convenience. Graph queries scale differently from join-heavy SQL. Adding more relationships does not degrade traversal performance the way adding more joins degrades query plans. For an agent memory that grows over months of use, that scaling property matters.
Neo4j's 2025 work on Graphiti - a real-time knowledge graph engine for agent memory - confirmed the architecture decision. Graphiti uses a three-memory-type model:
- Short-term memory: conversation history in working context
- Long-term memory: entity knowledge graph with temporal awareness
- Reasoning memory: decision traces and provenance records
The temporal awareness is particularly useful for desktop agents. Not just "the user uses Notion" but "the user switched from Notion to Linear in October, with the transition happening over three weeks, and they still use Notion for personal projects." That time-aware relational structure is natural in a graph database and awkward in a flat schema.
Building It with Claude Code
Building an agent with Claude Code is recursive - using an AI coding agent to build an AI agent. The experience is instructive about where AI-assisted development works well.
Claude Code understands Go idioms. It generates correct goroutine patterns, uses channels appropriately, and avoids common Go pitfalls like goroutine leaks. It generates valid Cypher queries for Neo4j that handle the common edge cases around null relationships and empty result sets. It handles the boilerplate of macOS desktop integration - accessibility permissions, menu bar setup, system notifications - while you focus on the architecture decisions.
A practical example: setting up the goroutine architecture for concurrent screen watching and user input handling. The initial implementation had a channel deadlock in the shutdown sequence - a classic Go mistake where two goroutines are each waiting for the other to send before either exits. Claude Code diagnosed the issue from the log output and generated a corrected implementation using a context-based cancellation pattern in about two minutes.
// Context-based shutdown pattern - avoids deadlock in goroutine cleanup
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
go screenWatcher(ctx, eventChan)
go inputProcessor(ctx, eventChan, memory)
// Wait for interrupt signal, then cancel context - both goroutines exit cleanly
sig := make(chan os.Signal, 1)
signal.Notify(sig, syscall.SIGINT, syscall.SIGTERM)
<-sig
The result is a functional alternative architecture built in weeks instead of months - because the boilerplate problem, which is most of the work in building a new desktop agent, is solved by the coding agent.
When to Choose This Architecture
Go + Neo4j makes sense when:
- You need true concurrency across many simultaneous operations
- Your agent's memory needs to represent rich relationships between entities
- You want a single deployable binary with no runtime dependencies
- Performance and memory efficiency matter more than access to Python's ML ecosystem
Python + SQLite or markdown makes sense when:
- You need to use Python ML libraries directly in the agent
- The memory structure is simple (flat preferences, recent conversation history)
- Setup simplicity matters more than runtime performance
Neither architecture is universally correct. The choice depends on what your agent needs to do.
Fazm is an open source macOS AI agent. Open source on GitHub.