Local AI hardware tradeoffs on Apple Silicon: bandwidth, memory, and the third axis no one mentions

Honest answer: it depends on what you mean by local AI. For chat-only LLM inference, memory bandwidth decides decode speed and the order is base 120 GB/s, Pro 273 GB/s, Max 546 GB/s on M4. A Max generates roughly 4.5x faster on the same model than a base. For a desktop agent that drives apps and the browser, the bottleneck flips to prefill compute and to per-turn input size, and a Pro tier is often enough if your agent uses an accessibility tree instead of feeding the model a full screenshot every turn. For diffusion image generation, total unified memory matters more than bandwidth because the bottleneck is fitting the model and the latents, not streaming weights per token.

Because LLM decode (the per-token generation phase) is memory-bandwidth bound. Each new token requires streaming the entire weight matrix through the chip. Bandwidth in GB/sec divided by model size in GB gives a hard ceiling on tokens per second. That is true and the existing buyer's guides cover it well. What they leave out is that the decode phase is only one of three things a real local AI workload spends time on. The other two are prefill (processing the input prompt before the first token) and tool execution (everything that happens between turns of an agent loop). Once those enter the picture the chip choice changes.

Decode is one weight pass per token output. Prefill is one weight pass plus matrix multiplications across the entire input length. On a long input, prefill is the wall you hit before the model emits anything. On Apple Silicon, prefill scales with GPU TFLOPS (compute), not with memory bandwidth. The Max has more cores than the Pro and the Pro has more than the base, which is why the bandwidth ranking and the prefill ranking happen to look similar, but they are not the same number. A 10,000 token input on a 13B model on an M4 base is dominated by prefill time. A short chat reply on the same chip is dominated by decode.

Chat is one round trip. The system message is small, the user message is small, the model writes a few hundred tokens, done. An agent is a loop. Each turn re-sends the system prompt, the tool schemas, the conversation history, and a fresh dump of the screen state. After three or four turns the per-turn input lives in the 10,000 to 15,000 token range. With prefill running at a few hundred tokens per second on consumer hardware, that is many seconds of waiting before the model can pick its next action. Multiply by the number of turns it takes to finish a task and you get the difference between an agent that feels responsive and one that you give up on.

There are two ways to tell a model what is on screen. One is to send a screenshot to a vision-capable model, which costs roughly 1,500 to 3,000 input tokens per image depending on resolution and tile policy. The other is to walk the macOS accessibility tree and serialize each visible element as a small object (role, optional text, optional bounding box). A serialized accessibility tree of the same window typically lands in the 200 to 400 token range. Across a ten-step task the difference compounds to roughly 6x the cumulative input tokens. That 6x sits on top of every other hardware choice you make. If you pick the screenshot path, you have effectively undone the bandwidth advantage you just paid extra for.

Yes, if the agent is built around an accessibility tree rather than a screenshot. An M3 Pro or M4 Pro has enough prefill compute to chew through a 2,000 to 3,000 token per-turn input in a few seconds, and the decode at 25 to 35 tokens per second is comfortable for short tool-call emissions. The same Pro chip falls over running a screenshot-based agent because each turn balloons to 10,000 plus tokens of input and the prefill cost scales linearly. The hardware did not change. The representation did.

Settings > AI Chat > Custom API Endpoint. The text field writes to a UserDefault called customApiEndpoint at SettingsPage.swift line 885. When the agent subprocess spawns, ACPBridge.swift lines 468 to 470 reads that value and exports it as ANTHROPIC_BASE_URL into the bridge environment. Any local runtime that exposes an Anthropic-compatible Messages API works: a small shim in front of llama.cpp's server, an Anthropic mode on vLLM, an OpenRouter Anthropic proxy, LM Studio with the right plugin. The agent code does not change. Only the URL does.

Different bottlenecks for each. Diffusion image generation is bound by total unified memory plus GPU compute, because the model and the latent buffers all need to live in RAM and the per-step work is matmul-heavy. Bandwidth matters less than for LLMs. Video generation is dominated by VRAM (so unified memory size) and by sustained thermal headroom; this is a Max or Studio category problem. Whisper and other ASR models are small and prefill bound on a single audio buffer, so any current Apple Silicon chip handles them in real time. Lumping all of these into a single 'local AI' bucket and asking which chip is best is what makes most buyer's guides unhelpful.

For models that fit in consumer GPU VRAM (say up to a 13B at 4-bit on a 16 GB card), no. A discrete card with 1,000 plus GB/sec of bandwidth runs them faster. For models that do not fit, Apple Silicon wins by being able to load them at all. A 70B at 4-bit needs roughly 40 GB of memory and there is no consumer single-GPU answer to that other than a Mac with 64 plus GB of unified memory. The honest framing is not better or worse but coverage of the model size range you actually plan to run.

Per-turn end-to-end latency. Take your typical per-turn input size in tokens, divide by your runtime's measured prefill speed, add a typical decode time for the tool-call output, then add tool execution time. If that lands under five seconds you have a snappy agent. Five to fifteen seconds is workable but you will notice. Over fifteen seconds and people stop using it. If the prefill term dominates and you cannot change chips, change the screen-state representation before you change anything else.