LLM quantization updates 2026

MXFP4 picking up real hardware support. The Open Compute Project's microscaling FP4 format used to be a software-only idea on consumer machines. With Apple's A19 Pro the MXFP4 path has dedicated silicon, which collapses the dequantization overhead that made earlier 4-bit floating-point formats slower than INT4 on consumer Apple chips. That hardware path makes MXFP4 a serious option for an on-device runtime in a way it was not in 2025.

Block size and scaling. MXFP4 (the OCP standard) uses 32-value blocks with a single E8M0 power-of-two scaling factor per block. NVFP4 (Nvidia's Blackwell-native format) uses 16-value blocks with a two-level scaling strategy: a fine-grained E4M3 scaling factor per block plus a second-level FP32 tensor-wide scalar. Smaller blocks plus the higher-precision scale mean NVFP4 retains a bit more accuracy, at the cost of needing actual Blackwell tensor-core dispatch to run efficiently. Per the Nvidia developer blog, NVFP4 cuts memory footprint 3.5x versus FP16 and 1.8x versus FP8 with under 1 percent accuracy loss on language tasks.

Usable, not a drop-in replacement. Microsoft's BitNet repo on GitHub is MIT-licensed and ships an inference framework. The catch is that 1.58-bit weights have to be trained that way from scratch, which means you do not get to take an existing 70B model and crush it down. The largest publicly trained BitNet b1.58 model is 8B (BitNet b1.58 2B4T). At 2B parameters trained on 4T tokens, it benchmarks within 1 to 2 points of full-precision peers on MMLU, GSM8K, HumanEval+, and reduces footprint from roughly 2 GB to 0.4 GB. That is real. It is also still capped at the 8B scale ceiling.

It pushes the KV cache, not the weights, down to extreme quantization levels. KV-cache memory is what blows up first as you scale context length, and weight quantization does nothing to fix it. TurboQuant (Zandieh et al., ICLR 2026) is being prototyped against llama.cpp via Discussion #20969 and is the first KV-quantization scheme aggressive enough to consider for consumer hardware running long agent sessions. If you have ever watched a local agent OOM after the conversation grows past a few thousand turns, this is the lever.

Same algorithm, much faster calibration. The 0.9.0 release of LLM Compressor added batched calibration for AWQ, with batch_size=32 yielding roughly a 3x speedup on large models because AWQ relies on many onloaded forward passes during calibration. AWQ remained the default INT4 production-inference format through 2025, and faster calibration is what made it tractable to recompute scales for new fine-tunes without renting a small cluster.

Fazm has a Custom API Endpoint setting in Settings, AI Chat, AI Chat. The toggle and text field are defined in SettingsPage.swift lines 954-999 with placeholder 'https://your-proxy:8766'. The value is stored as the customApiEndpoint UserDefault and exported as ANTHROPIC_BASE_URL on the bridge subprocess at ACPBridge.swift lines 468-469. Anything that speaks Anthropic's Messages API can sit at that URL: a llama.cpp server with an Anthropic shim, an MLX server hosting a 4-bit group quantized model, an Ollama-fronted GGUF, or an OpenRouter Anthropic mode endpoint. Fazm itself does not ship a quantization engine. It ships the hook that points the agent at one.

Not yet, and the dominant production usage on consumer Macs is still GGUF k-quants (Q4_K_M, Q5_K_S, Q6_K). The k-quants format mixes integer block quantization with per-block scales and minor outlier handling and remains the default download target on Hugging Face for community fine-tunes. The 2026 movement is layered on top: AWQ scales applied before GGUF quantization to protect salient weights, and FP4 formats (NVFP4, MXFP4) rolling in for hardware that supports them. For most Mac users in mid 2026 the practical answer is a Q4_K_M or Q5_K_M GGUF on llama.cpp, with FP4 reserved for users on Apple A19 Pro or Blackwell discrete GPUs.

MLX uses 4-bit group quantization as its native format, optimized for Apple's unified memory architecture. There is no MLX-specific format announcement in 2026 in the same sense as NVFP4 from Nvidia. The interesting MLX side is that MXFP4 with hardware support on A19 Pro now competes with MLX's own 4-bit group format, and the open question is whether Apple's stack will adopt MXFP4 as a first-class quantization target or keep MLX's group quantization as the default.