The number that matters for a local LLM desktop agent is not the one in your chat benchmark

Why chat benchmarks mislead you about agent throughput

Chat is one prompt followed by one stream of tokens. You see the first token, you start reading, and the visible throughput is whatever the runtime emits per second after prefill ended. Prefill happens once and the user does not feel it on a short input.

An agent is a loop. Every turn the runtime re-receives a prompt that contains the system message, the full tool schema, the message history so far, and the latest snapshot of the world. The runtime then runs prefill across that entire input from scratch (modulo prefix caching, which most consumer setups do not have configured), produces a small JSON tool call, your runtime executes that tool, the result is appended, and the loop runs again. Every iteration repeats the prefill cost.

On consumer Apple Silicon the relevant numbers look roughly like this: a 13B class quantized model on an M3 Pro decodes at 15-22 tok/s, while prefill on the same machine usually lands somewhere between 150 and 400 tok/s depending on backend and quantization. On an M4 Max with 64-128 GB you can push prefill higher, but the ratio of prefill speed to decode speed stays in the same ballpark across the consumer range.

Now imagine a turn that re-reads 12,000 input tokens and emits 150 output tokens. At 200 tok/s prefill and 18 tok/s decode that is 60 seconds of prefill and 8 seconds of decode. The user feels a 68-second wait. Halve the input and the prefill term collapses faster than anything you can do to the model.

Screen state is the dominant variable, not model size

The agent has to know what is on screen to decide what to do next. There are two ways to give the model that information.

One: take a screenshot, send it as an image. A vision-capable model tiles that image and produces input tokens for each tile. A 1440x900 region typically lands in the 1,500 to 3,000 token range. The model has to attend over every pixel: text, icons, decoration, ads, scrollbar chrome, the OS menu bar. None of that is interactable, but all of it consumes the prefill budget.

Two: walk the accessibility tree of the focused window and serialize it. macOS exposes a structured representation of every visible interactable element through the AXUIElement API. The model receives a JSON array of objects with one role, an optional label, and a four-number bounding box per element. A typical app screen serializes to 200-400 tokens. There is nothing about decoration, no rendering chrome, no menu-bar noise unless you ask for it.

Over a 10-step task the public Fazm benchmark on browser-tool comparison measured a 150,000 vs 25,000 input-token gap between the two approaches. Six times. That gap rides on top of every other optimization you do. Speculative decoding and prefix caching help the screenshot path too, and the screenshot path is still the slow one because the input was always larger.

The accessibility tree shape, with the actual cap constants

The exact shape Fazm uses comes from the open-source MacosUseSDK. The traversal lives in Sources/MacosUseSDK/AccessibilityTraversal.swift. Three caps keep the per-call output bounded:

• maxElements = 2000, past which the traversal returns with truncated = true in stats.
• maxDepth = 100, past which child recursion stops.
• maxTraversalSeconds = 5.0, past which whatever was collected is returned with a warning.

The element shape itself is six fields:

public struct ElementData: Codable, Hashable, Sendable {
    public var role: String
    public var text: String?
    public var x: Double?
    public var y: Double?
    public var width: Double?
    public var height: Double?
}

That is the entire payload per element. No parent pointers, no styling, no z-index, no DOM-style attribute bag. Most elements serialize to 30-60 JSON tokens; the median screen lands well inside the 200-400 token band even when the underlying app has a deep view hierarchy. The cap constants exist precisely so a pathological app cannot blow up the prefill budget on you.

How a single turn actually runs against a local LLM

The wiring inside Fazm: when you toggle Custom API Endpoint in Settings (SettingsPage.swift line 887, the textfield at line 983), the value lands in a UserDefault called customApiEndpoint. When the agent process spawns, the bridge layer (ACPBridge.swift line 406-408) reads it and exports it as ANTHROPIC_BASE_URL in the agent's environment.

From that point the agent does not know it is talking to a local model. It calls the Messages API, the call hits whatever URL you pointed at: an Anthropic-compatible vLLM server on localhost:8000, an LM Studio instance behind a small shim, the Z.ai Anthropic-compatible endpoint, GitHub Copilot's Anthropic bridge. The reasoner is hot-swappable through one env var. The agent's tool surface (browser via Playwright extension, Mac apps via the macos-use MCP, terminal, file system, Google Workspace) stays identical.

Prefix caching is the second lever, and it is free

Most of the per-turn input does not change. The system prompt is identical across turns. The tool schemas are identical. The first N messages of the conversation are identical. Only the trailing piece (the latest tool result + the new accessibility tree) is new.

Runtimes that support automatic prefix caching skip the prefill cost on the unchanged part of the prompt. llama.cpp via cache_prompt, vLLM via its automatic prefix cache, MLX servers via their KV reuse hooks. Turn it on, configure your model server to keep enough KV memory available for the active session, and your effective per-turn prefill collapses to the size of the diff between turn N and turn N+1. With the accessibility-tree path, that diff is small because the tree is small to begin with.

On the screenshot path, prefix caching helps less. Each new screenshot is a different image, and image tokens after re-tiling do not align with the previous turn's image tokens, so the cache does not get a hit on the changing piece. You eat full image-token prefill on every turn.

What actually fits on consumer Apple Silicon

The realistic operating envelope, assuming Anthropic-compatible serving and tool-use-trained models:

• M2/M3 base, 16 GB: 7B-class instruct at 4-bit, decode 25-40 tok/s, prefill 200-400 tok/s. Fine for one-app workflows where the tree stays small. Will choke on long task histories without aggressive context trimming.
• M3 Pro, 18-36 GB: 13B-class instruct at 4-bit, decode 15-22 tok/s, prefill 150-300 tok/s. The practical floor for cross-app workflows. With prefix caching on, per-turn latency lands in the 5-8 second range.
• M4 Max, 64-128 GB: 30-70B at 4-bit, decode 30-45 tok/s, prefill 400-800 tok/s. The tool-call quality is what makes this band actually useful, not the throughput. The throughput is fine for any honest desktop-agent task at this size.

Below the M3 Pro band, the limiter usually flips from throughput to tool-call accuracy: the small models miss the right element to click. That is a model-quality problem, not a throughput problem, and no amount of optimization fixes it.

What the runtime actually prints when you watch one turn

A representative turn against a 13B class model behind an Anthropic shim, with the accessibility-tree path on, looks something like this in the runtime log:

Two things to notice. First, the cache-hit term is most of the prefill budget. Without prefix caching the same turn would have eaten 11,847 / 218 = 54 seconds of prefill instead of 6. Second, the tree column is 307 tokens. If that line said 2,400 because we sent a screenshot, prefill would be 11 seconds even with the cache hit, and on a fresh conversation (no cache) we would be back at the minute-plus regime.

The actual operating checklist

If your local desktop agent feels slow, work the list in this order:

1. Audit per-turn input length. Log input_tokens on every Messages API call. If it is over 15K, you have a screen-state representation problem before you have a runtime problem.
2. Switch from screenshots to accessibility tree for any agent step where the model only needs to know about interactable elements. Reserve screenshots for genuine vision steps (reading a PDF, identifying a chart, OCR fallback when accessibility fails).
3. Turn on prefix caching in your runtime. llama.cpp cache_prompt: true, vLLM --enable-prefix-caching. Confirm cache hits in the runtime log.
4. Trim tool schemas you do not use. A typical agent ships 30+ tools and the model only needs 3-5 for any given session. Tool schemas are deceptively heavy because every parameter description costs tokens.
5. Cap conversation history. Past 10-15 turns the model rarely benefits from the full transcript. Summarize or window aggressively.
6. Only then look at the model. Quantization choice, decode tok/s, speculative decoding. These all matter, but in the agent regime they fight for the small slice of the latency budget that decode actually owns.

The cases where this rule breaks

Two honest counterexamples. First, on canvases where the accessibility tree is genuinely thin (Figma, Photoshop, Blender, custom Electron apps that ship no AX hierarchy), the tree is unhelpful and a screenshot is the only path. The argument here only applies when the app actually exposes its semantic structure.

Second, for short tasks (under 3 turns), the prefill cost is paid only a few times and the difference between 2 seconds and 12 seconds of total prefill is annoying but not disqualifying. The argument compounds with task length, and most useful desktop-agent tasks are 10+ turns.

Third, with a real frontier-class hosted model, prefill is so fast that the screenshot/tree distinction matters far less for latency, though it still matters for cost. The local-LLM regime is where the math tilts hardest.