Built a Local AI Coding Agent with Qwen 3.5 9B

Fazm Team··2 min read

Built a Local AI Coding Agent with Qwen 3.5 9B

Running a coding agent entirely on your own hardware sounds impractical until you try Qwen 3.5 9B on an M-series Mac. The model fits comfortably in memory, responds in under two seconds, and handles tool calling well enough for real desktop automation.

Why Tool Calling Format Matters

The biggest lesson from building this was not about model quality - it was about tool calling format. The same model can go from useless to reliable just by changing how you structure the tool definitions in the system prompt.

Qwen models expect tool calls in a specific JSON format. If you use OpenAI-style function calling syntax, the model gets confused and starts hallucinating tool names. If you match Qwen's native format, tool calls become consistent and parseable.

The format looks like structured XML-ish blocks with clear parameter definitions. Once you get this right, the 9B model calls the correct tools about 85% of the time - not perfect, but good enough for supervised automation.

Local LLM for Desktop Automation

The real advantage of running locally is latency and privacy. Every tool call round-trips in milliseconds instead of the 500ms-2s you get with cloud APIs. For desktop automation where the agent might make 50 tool calls to complete a task, that adds up fast.

Privacy is the other win. Your screen contents, file contents, and workflow patterns never leave your machine. For anyone working with sensitive data, this is not a nice-to-have - it is a requirement.

The Practical Setup

Run the model through Ollama, connect it to MCP servers for file system and accessibility API access, and add a simple loop that parses tool calls and executes them. The whole system is surprisingly straightforward once the tool calling format is dialed in.

The 9B model is the sweet spot for M-series Macs with 16GB RAM. Big enough to reason about code, small enough to leave room for your actual apps.

Fazm is an open source macOS AI agent. Open source on GitHub.

More on This Topic

Related Posts