llama.cpp release May 2026, read from inside a Mac agent

Roughly fifty builds, from b9070 on May 8 through b9127 on May 12. Most of them are GPU-backend or model-architecture work that does not touch a Mac path (Adreno q4_0 and q4_1 MoE for OpenCL, SYCL flash-attention allocation, Hexagon HVX kernels, Vulkan shader fixes). Four changes do matter for a Mac AI agent: b9077 (server gains a Vertex AI compatible API), b9100 (sampling can return post-sampling probabilities), b9101 (server prints a warning when an HTTP timeout is exceeded instead of hanging), b9109 (parallel drafting support for speculative decoding), b9114 (metal: promote mul_mv and mul_mm batch divisors to Metal function constants), and b9124 (mtmd, server, common: expose modalities at /v1/models so clients can feature-detect vision and audio before sending a request).

It means the integer divisors used inside the matrix-vector and matrix-matrix Metal kernels are no longer plain runtime arguments. They are now Metal function constants, which the Metal compiler specializes at pipeline-creation time. The result is a smaller hot-path: the divisor becomes a compile-time integer constant inside the kernel, so the GPU can fold the division into shifts where the divisor is a power of two and skip a load otherwise. For a Mac agent driving local llama-server, this lands on the prefill path that handles your accessibility-tree system block every turn. You will not see a graph-redrawing speedup on the M1 generation, but on M3 and M4 the practical effect is a few percent off prefill latency on long-context turns. Compared with the Hexagon and OpenCL work in the same week, this is the only change that hits Apple Silicon at all.

It exposes, on the same /v1/models endpoint clients already poll, which input modalities each loaded model supports. So a client can ask the server, before sending a request, whether the model accepts text only, text plus images, text plus audio, or all three. For a Mac agent that ships a single binary and lets the user point it at any llama-server, this is the difference between sending a tool call with a base64 screenshot and getting a 400 because the model is text-only, vs. detecting modalities at startup and either degrading gracefully or surfacing a clear message. Fazm uses accessibility-tree text, not screenshots, so the modality question rarely bites in normal operation, but the moment the user wires a vision-capable shim in or hands the agent an image-bearing PDF, this matters.

It adds a Vertex-AI-shaped surface to llama-server, mirroring what they already do for the Anthropic and OpenAI shapes. If your Mac agent's API shim happens to speak Vertex (because it was built originally for a Google Cloud workload), you can now point that shim directly at llama-server without a third-party translator in the middle. Concretely, this matters for teams who run a Vertex AI workload in production, want to dogfood the same shape against a local llama.cpp instance for development, and only have a Mac. It does not change the Anthropic path, which is what Fazm's ACP bridge actually uses, so the headline is 'one fewer reason to stand up LiteLLM in front of llama-server'.

Before b9101, when llama-server's internal HTTP timeout fired on a slow generation (think a long decode on a 70B at Q4 on an M2 Air), the server would just hang from the client's perspective. After b9101, it logs a visible warning the instant a timeout is exceeded. For a Mac agent driving an Anthropic-shaped shim that fronts llama-server, this is the difference between 'why did the chat freeze at turn 12' and 'oh, look in the server log, there is the timeout'. The change is in the logging layer, not the behavior, but it cuts the debugging loop for the most common Mac-side failure mode of a local llama-server setup roughly in half.

Fazm reads one UserDefaults key, called customApiEndpoint, inside Desktop/Sources/Chat/ACPBridge.swift. When the value is non-empty, the ACP bridge subprocess is spawned with ANTHROPIC_BASE_URL set to it. That is the entire integration surface for swapping the backend, and it lives at lines 467 through 470 of ACPBridge.swift. The four lines do not care which version of llama.cpp is on the other side, only that whatever the URL points at speaks the Anthropic Messages shape. So every change in this May 2026 build window either flows through transparently (b9114 Metal speedups, b9100 sampling probabilities) or shows up the moment your shim asks llama-server for them (b9124 modalities, b9077 Vertex shape).

Two reasons to consider it. First, b9114 is a small but real Metal prefill-path improvement, and it has zero cost to take. Second, b9101's timeout warning will save you the next time the server stalls under load and you cannot tell whether the bottleneck is the model or the network. The rest of the May builds are mostly non-Apple paths (Adreno, SYCL, Hexagon, Vulkan). If you are on a build older than April's b8920 (the metal-print-GPU-description change), you also lose a useful boot-time sanity log line by not upgrading. None of the May changes break the Anthropic shape that a Fazm-style agent depends on.

Only if your shim actually exposes the speculative-decoding knob to the model loader. The May change enables multiple draft models to run in parallel during speculative decoding, which trades a bit of memory for higher draft acceptance. On a Mac, the constraint is almost always unified-memory bandwidth, not draft throughput, so the win is smaller than on a discrete-GPU host. If you are loading a 30B target plus a 1B draft on a 64 GB M3 Max, you can probably fit two drafts and see a measurable improvement on tool-call-heavy turns. On a 36 GB M3, you will run out of memory before you see the gain. Treat it as an advanced setting, not a default.

Because Apple does not ship Vulkan, and the WebGPU path inside llama.cpp targets browsers, not native applications. A signed and notarized Mac agent talks to llama-server over HTTP through the Metal backend, full stop. The Vulkan Windows-Intel BF16 regression fix in b9119 is genuinely useful, just not on this platform. Same for the OpenCL Adreno work, which targets mobile GPUs.

Three steps. One, build or pull llama.cpp at tag b9127 or later, run `llama-server -m <model> -c <ctx> --host 127.0.0.1 --port 8080`. Two, drop an Anthropic-shape shim in front, the simplest path is `litellm --config litellm.yaml --port 4000` with a single model entry mapping `anthropic/claude-sonnet` to your local OpenAI-shape llama-server. Three, install Fazm from fazm.ai, open Settings, click Advanced > AI Chat > Custom API Endpoint, paste http://127.0.0.1:4000 and save. The four lines at ACPBridge.swift:467-470 pick the URL up on the next chat turn and your local b9127 server is in the loop. Verify by checking llama-server's stdout the moment you send a message: you should see the request hit and the Metal GPU description line print (from April's b8920).

It changes which gains you actually feel. Screenshot-based Mac agents put a vision-encoder step in front of every turn, so they care intensely about whether b9124's modality exposure matches their model's image input contract, and they pay the encoder latency even when nothing visual changed on screen. An accessibility-tree agent sends 2 to 10 KB of structured text per turn, mostly stable across turns, which is exactly the traffic pattern b9114's Metal mul kernel and the Anthropic prefix-cache work from April benefit. The same llama.cpp build that helps both agent shapes a tiny bit on raw decode helps the accessibility-tree shape a lot on prefill, because the prefill is what dominates the turn.