Local LLM runtime done, agent loop missing

Because the runtime is one piece of the system, not the whole system. Ollama, LM Studio, llama.cpp, vLLM and friends are model servers. They take a request, run a forward pass, and stream tokens back. They do not define tools the model can call, do not execute those tools, do not maintain conversation state across turns, do not represent screen state, do not schedule recurring runs. An agent loop is the code that does all of those things and uses the runtime as one component, the way a web app uses a database. Loading a model and waiting for an agent to emerge is like installing Postgres and waiting for an app to appear.

A program that calls the model in a loop, passing back the output of the previous tool call as the next user turn, until the model emits a stop signal. That is it. Everything else (tool definitions, tool sandbox, state, memory, scheduling) is plumbing the loop needs to actually do useful work on a real machine.

Two things outside the runtime. First, a tool schema sent to the model as part of the prompt, normally as a JSON schema that names each tool, its arguments, and what it does. Second, a parser on the host side that recognizes when the model has emitted a tool-call structure and dispatches it to a real function. The runtime just streams tokens. The schema design and the dispatcher live in the agent loop. In Fazm's open source bridge those 19 stdio tools are defined at acp-bridge/src/fazm-tools-stdio.ts (execute_sql, capture_screenshot, request_permission, scan_files, ask_followup, routines_create, and so on), and the dispatcher is wrapped by the watchdog in acp-bridge/src/index.ts around lines 109 to 240.

Yes, and that is the cleanest way to fill the gap. The seam to look for is whether the framework lets you swap the model endpoint without touching the rest. In Fazm that seam is one Settings field that writes to UserDefaults key customApiEndpoint at SettingsPage.swift line 885, and one env-var assignment at ACPBridge.swift line 469 that exports it as ANTHROPIC_BASE_URL when the bridge spawns. Anything that speaks the Anthropic Messages API plugs in there: an LM Studio shim, a vllm-mlx server, an Anthropic-compatible proxy. The loop and the 19 tools stay constant; only the reasoner behind them changes.

Because most desktop work is not in the chat window. It is in another app, behind a button the model has never seen, on a screen that is changing every few seconds. The loop needs a way to convert the current screen into a structure the model can reason about and reference by ID when it emits a click. The two real options are a screenshot (vision-tokenized, roughly 1500-3000 input tokens per turn) or a compact accessibility tree (200-400 tokens for the same window, with explicit roles and bounding boxes). The accessibility-tree path is what makes the loop affordable on a local 13B model; the screenshot path is what every framework defaults to and why local agents feel slow.

Three layers most setups conflate into one. First, the in-flight message list the model sees on this turn (system prompt + tool schemas + last N turns + new user input). Second, the persisted thread that survives a process restart (a SQLite or file-backed store, so closing the app and reopening it does not lose the conversation). Third, longer-lived facts the loop reloads at the start of every session (a memory file like MEMORY.md plus topic files, so the model knows who the user is across sessions without burning tokens on every turn). A bare runtime owns none of these. The loop has to.

Most useful agent work is not one-off. It is running every morning at 9 to summarize unread Gmail, or every hour to check a Reddit thread, or every Friday to invoice clients. A scheduler that fires the loop on a cron or a calendar trigger is part of an agent system, not a plain LLM app. Fazm's bridge ships a runner at acp-bridge/src/cron-runner.mjs and exposes routines_create, routines_list, routines_update, routines_remove, and routines_runs as tools so the agent itself can schedule its own future runs. The runtime knows nothing about any of that.

MCP (Model Context Protocol) is a standard for exposing tools to a loop, not a replacement for the loop. An MCP server like macos-use, playwright, or fazm_tools defines a set of tools and a transport (stdio, HTTP, websocket). The loop has to spawn it, register the tools, route tool calls into it, surface the response, and time it out if it hangs. In Fazm's bridge, MCP tools have a 120s timeout and other tools have a 10s or 300s timeout (TOOL_TIMEOUT_INTERNAL_MS, TOOL_TIMEOUT_MCP_MS, TOOL_TIMEOUT_DEFAULT_MS at acp-bridge/src/index.ts lines 114-116). Without those timeouts a single hung subprocess freezes the agent. MCP is one piece, not the whole.

Two on macOS that are worth opening side by side. Fazm at github.com/mediar-ai/fazm wraps Anthropic's claude-agent-acp 0.29.2 (acp-bridge/package.json) and adds 19 native tools, a permission system, a scheduler, screen-state via accessibility APIs, and a customApiEndpoint seam for swapping in any Anthropic-compatible local runtime. Cline and Goose are also good references for the same shape on a different transport. Read whichever is closest to what you want; once you have seen the loop, the runtime stops feeling like the bottleneck.

Three things and only three things. The forward pass (tokens in, tokens out). The KV cache and any prefix caching tricks that make repeated turns cheap. The decoding strategy (sampling, speculative decoding, structured output mode, JSON mode). Everything else (when to call the model, what to put in the prompt, how to parse the output, how to react to it, what to remember) belongs to the loop. Picking a runtime is mostly about prefill speed and prefix caching support; picking a loop is about the other five pieces on this page.