LLM quantization 2026 updates, judged by whether the model can still press the right button

The headline change is that FP4 (four-bit floating point) went from research to a thing you can run. Two formats matter: MXFP4, the Open Compute Project microscaling standard backed by AMD, ARM, Intel, Microsoft, Meta, and Qualcomm, and NVFP4, NVIDIA's higher-fidelity variant. Both encode each weight as E2M1 (1 sign, 2 exponent, 1 mantissa) but differ in block size and scaling: MXFP4 uses 32-element blocks with a single E8M0 scale, NVFP4 uses 16-element blocks with a two-level scale (FP8 E4M3 per block plus an FP32 per-tensor factor). NVFP4 support merged into llama.cpp through a run of PRs from late March through April 2026 (the CUDA dp4a kernel on March 26, Vulkan support April 10-14, Blackwell-native tensor-core dispatch by late April), per InsiderLLM's tracking of the merges. The practical payoff is size: a Qwen 3.6-27B drops from roughly 17GB in Q4_K_M to about 14GB in NVFP4. Everything else in 2026 quantization is a refinement of methods (GPTQ, AWQ, QAT) feeding into those formats.

Because an agent's output gets executed, not read. When Fazm runs a turn, the model does not produce prose for a human to skim. It produces a tool_use block: a function name and a JSON argument object that the macos-use layer turns into a literal click, keystroke, or scroll against your real Mac. If quantization nudges one numeric argument (an element index, an x/y coordinate, a row number), the result is not a slightly worse sentence. It is the wrong button pressed. A chatbot can absorb a small drop in fluency and the reader never notices. An agent cannot absorb a malformed or mis-targeted tool call. So the metric that should govern your quant choice for agent work is tool-calling validity, not perplexity or MMLU, and that is exactly the number the format spec sheets never report.

It is promising but unproven for this specific use. The honest state of things as of mid-2026, in InsiderLLM's words, is that 'no independent benchmark suite has run NVFP4 GGUFs against Q4_K_M GGUFs on a representative model set yet.' Early community reports suggest FP4 can be worse than Q4_K_M on smaller models, where there is less redundancy to absorb the precision loss. For agent work, where tool-call validity is fragile, that uncertainty is a reason to wait for evidence rather than chase the size win. The conservative 2026 choice for a local agent brain is still a well-made 4-bit integer quant (Q4_K_M or a 4-bit MLX build on Apple Silicon). Treat FP4 as a size optimization to test on your own tool suite before you trust it to click around your apps.

Four bits, in practice. The 2026 tool-calling evaluation by JD Hodges held every one of 13 models at Q4_K_M and still saw a spread from 97.5% pass rate (Qwen3.5 4B) down to 15% (xLAM-2 8B), which tells you that at Q4 the bottleneck is the model's training and architecture, not the quantization. Drop below four bits (Q3, Q2) and you add a second source of failure on top of that: the quant itself starts corrupting the structured output. The exception is quantization-aware training (QAT), where the model was trained to be run at a specific low precision. In that same eval, Gemma 3 4B QAT was run at Google's pre-quantized Q4_0 precisely because QAT means the weights expect it. Outside QAT, four bits is the floor for agent reliability.

Not by default, and that is the point of the distinction. Fazm's default brain is a frontier hosted model: Claude Code, or Codex, reached through the Agent Client Protocol. Those are full-precision cloud models with millions of tool-use trajectories behind them, which is why tool calls are reliable out of the box. Quantization only enters the picture if you choose to point Fazm at a local quantized model. The hook is one block in Desktop/Sources/Chat/ACPBridge.swift: if the customApiEndpoint setting is non-empty, Fazm writes it to ANTHROPIC_BASE_URL on the agent subprocess. Front a local NVFP4 or Q4_K_M model with an Anthropic-compatible shim, paste the URL, and the same agent loop reasons with weights that never leave your Mac. The quantization-quality question above is then yours to own.

Run the quantized model under a runtime that can serve it (Ollama, mlx-lm, LM Studio, or llama.cpp's server), put a shim like LiteLLM in front to translate the runtime's API into the Anthropic Messages shape, then open Fazm > Settings > AI Chat and set Custom API Endpoint to the shim's local URL (for example http://127.0.0.1:4000). The deeper end-to-end walkthrough, including the request path for a single turn, lives in the on-device LLM updates guide linked at the bottom of this page. The quantization decision (which format, how many bits) is independent of the wiring: the endpoint field does not care whether the model behind it is FP16, Q4_K_M, or NVFP4, so the burden of picking a quant that survives tool calling is on you.

The NVFP4 and MXFP4 format details, the llama.cpp merge timeline, and the Qwen 3.6-27B size figures come from InsiderLLM's FP4-in-llama.cpp guide (insiderllm.com). The 13-model tool-calling pass rates, the Q4_K_M test setup, and the Gemma 3 QAT note come from JD Hodges' March 19, 2026 local-LLM tool-calling evaluation (jdhodges.com), run on LM Studio v0.4.6. The Berkeley Function-Calling Leaderboard (gorilla.cs.berkeley.edu) is the standard reference for measuring tool-call accuracy across models. Fazm's endpoint hook is in the open source at github.com/m13v/fazm under Desktop/Sources/Chat/ACPBridge.swift. Each is linked inline above.