The heterogeneous local AI scheduler gap is three gaps, and one tool will not close all three

Gap one, silicon: the ANE is idle and Metal is the default

On an M-series chip there are three compute blocks. The CPU has AMX matrix units that are good for small dense matmuls and prefill on short inputs. The GPU is the high-bandwidth target for streaming a large weight matrix through attention and feed-forward layers, and it is what Metal-backed inference uses. The Apple Neural Engine is the highest-throughput, lowest-power compute, but it expects CoreML-converted graphs with quantizations it supports, and it is dimensioned for vision and ASR shapes, not for streaming a 13B weight matrix per token.

A real silicon scheduler would split a single transformer graph across all three. The embedding and the small adapter passes go to the ANE. The big matmuls in attention and feed-forward go to the GPU. Some prefill bookkeeping goes to the CPU. Apple themselves ship their on-device foundation model with roughly this split. Third-party consumer runtimes do not. llama.cpp lives on Metal. MLX lives on Metal. Ollama is a wrapper around runtimes that live on Metal. The ANE goes idle. That is gap one.

MLX has experimental ANE paths. llama.cpp has a Metal-Performance- Shaders bridge that can route ops through CoreML. None of those are production for arbitrary models. The honest claim today is that the silicon scheduler is the runtime's job, and your job is to pick a runtime that is getting close, not to build the scheduler yourself inside an agent.

Gap two, work-type: an agent has five compute consumers, not one

Open a desktop agent and watch what it actually does in a minute of use. It reasons (forward pass on a big LLM). It transcribes what you said (ASR, ideally a streaming Whisper or Parakeet). It reads screen state (either OCR via Vision framework, or, better, the accessibility tree). It classifies your request to pick a route (small model, sometimes regex). It might embed something for memory (a small sentence-transformer). Five compute consumers. Five different ideal targets. Five different latency budgets.

Consumer local-AI tools treat this as one consumer because the local-AI conversation came from the local-LLM community, and a local LLM is one component. The work-type scheduler is the piece that says "transcription goes to a streaming endpoint, reasoning goes to messages.create, screen-state is the AX tree (no model at all), classification is a 3B local, embeddings are a CPU batch". That router is small. It is also missing from almost every local-agent tool I have looked at.

The interesting fact about this diagram is the second destination on the right. The accessibility tree is structured text that macOS already maintains for every running app. Reading it costs zero model inference. A serialized window of a typical macOS app lands in 200 to 400 tokens, versus 1,500 to 3,000 tokens for a screenshot fed to a vision model. The work-type scheduler that knows the AX path exists makes the silicon scheduler argument partly moot for this branch: you do not need to schedule a vision model across CPU/GPU/ANE if you do not run a vision model.

You can see this concretely in the open source bridge. The 19 stdio tools at acp-bridge/src/fazm-tools-stdio.ts include capture_screenshot as an option for the canvas/WebGL fallback case, but the default screen-state input to the model is the AX tree. The work-type router lives in the loop, not the runtime.

Gap three, quality vs latency: small model classify, big model reason

The user says "close the second tab". The right response is one tool call. The user says "summarize this 40-page PDF and email the gist to my accountant with a polite ask for a tax call". The right response is a long plan and several tool calls. Sending both to the same model is throwing capacity at the first one and starving the second one. Sending both to a small model fails on the second one. Sending both to a big model is slow on the first one and burns budget.

The classifier that picks the lane is the quality-vs-latency scheduler. It is also where most of the perceived speedup on a desktop agent lives, not in the underlying model inference speed. A 50ms classification that lets the loop short-circuit to a tool call without calling the big model at all wins on every UX metric a person can feel. The classifier itself can be local, on a 1.5B or 3B model, or even regex for the simplest cases. It is a different model from the reasoner. It is a different endpoint from the reasoner. It is missing in almost every consumer desktop agent, including the ones that beat their chest about being "fully local".

Counterargument: "just wait, Apple will ship it"

A reasonable reply is that Apple is the one party with the right incentives and the chip access to ship a real silicon scheduler, and they are obviously shipping pieces of it (the on-device foundation model, the new Vision Pro inference pipeline, the progressively richer CoreML toolchain). True, and irrelevant to the agent layer. Even when Apple ships a perfect silicon scheduler tomorrow, the work-type layer and the quality-vs-latency layer still need to exist somewhere. They are a different kind of code from a CoreML graph compiler. They live in the agent, not the chip.

The corollary is that an agent that decouples its model wire from its work-type router is not affected by which silicon scheduler you eventually settle on. The day a runtime ships true CPU+GPU+ANE co-scheduling, the agent points at that runtime via the same URL. The work-type and quality-vs-latency code underneath does not move. That is the design payoff for not building one monolithic local-AI agent.

Resolution: where the seam goes

Three layers, three places for the scheduler. Silicon scheduling belongs inside the inference runtime. It is the runtime's job to dispatch ops to CPU AMX, GPU Metal kernels, or the ANE for the parts of the graph that benefit. Pick a runtime that gets close. Work-type scheduling belongs inside the agent loop. The loop knows which compute consumer is firing on this turn, and it routes accordingly. Quality-vs-latency scheduling belongs in a top router that classifies the incoming request. It can be a regex, a 3B local, or a small cloud model.

In the Fazm codebase the seam between the agent loop and the runtime is two lines of Swift. The Settings UI writes a string; the bridge subprocess reads that string into an env var on spawn. Everything else (the 19 native tools, the cron scheduler, the accessibility-tree screen state, the conversation store, the permission system) sits above the seam and does not care what silicon scheduler runs on the other side of the URL.

// Desktop/Sources/MainWindow/Pages/SettingsPage.swift  (line 885)
@AppStorage("customApiEndpoint") private var customApiEndpoint: String = ""

// Desktop/Sources/Chat/ACPBridge.swift  (lines 467-470)
// Custom API endpoint (allows proxying through Copilot, corporate gateways, etc.)
if let customEndpoint = defaults.string(forKey: "customApiEndpoint"),
   !customEndpoint.isEmpty {
  env["ANTHROPIC_BASE_URL"] = customEndpoint
}

You will not find a scheduler in those two lines. That is the point. The scheduler is whatever is behind the URL, and the agent does not care. Anthropic's production API is one answer. A local mlx-omni-server with --use-ane is another. A llama.cpp build that just gained a CoreML branch is a third. The agent stays the same.

Practical reading order if you are designing one of these

Start with the work-type layer, not the silicon layer. Decide what the agent reads and writes per turn before you decide what chip block it runs on. The work-type layer is where the AX-tree decision lives, and that decision changes your per-turn token budget by roughly 6x. Once the work-type layer is honest, the silicon layer is a runtime choice you can make in the morning. Then add the quality-vs-latency router at the top, because that is where most of your felt latency goes. Do not start with chips. Start with the routing table.