Voice-first control of a laptop without sending audio to a third party: where 2026 actually lands.

The four layers, ranked by how local they already are

Three are answered. One is the bottleneck. Most arguments about "is local voice control viable" collapse all four into one and end up arguing about the wrong layer.

The four-knob gap, named precisely

The protocol layer is the only one where the local answer is "not yet". It is four specific controls. None of them is a model-quality question. All four are surface-area questions about what the streaming API exposes.

• interim_results. Partial hypotheses every ~100 ms while you are still talking. Local transducers can emit them. Public on-device SDKs (FluidAudio, WhisperKit) do not consistently expose a stable streaming partials API yet that matches what an agent UI needs.
• endpointing. A hard silence window that produces a speech_final event. Most local stacks ship voice activity detection (Silero, WebRTC VAD) as a separate library. Wiring it to the model so a single "turn-over" event lands at the right moment is the part every voice agent re-invents.
• runtime keyterm. A way to bias the decoder toward your contact list, app names, and file names this minute, without retraining. Server-side transducers expose this as a per-connection parameter. Parakeet TDT has no public runtime vocabulary boost in the open-source release. WhisperKit accepts an initial_prompt that biases output but does not boost specific terms.
• multichannel. Two raw PCM channels in one frame: channel 0 is your mic, channel 1 is the sound coming out of your speakers (the Zoom call, the meeting, the video). The result objects carry a channel_index so the agent knows whose words those were. Local SDKs accept a single audio buffer. If you want this split, you mux it yourself or run two model instances.

How Fazm actually splits this (the one cloud hop, named)

The Fazm Mac app runs locally for everything except the audio bytes. Screen reading uses AXUIElementCreateApplication against the focused window, in-process, no network. Clicks and keystrokes are CGEvent calls. The MCP servers run as child processes on your machine. The agent loop and the chat UI render locally. The only thing that leaves the laptop is the 100 ms PCM frame on the way to a streaming transducer, today Deepgram, because no open-source on-device stack reproduces the four-knob surface in May 2026.

The exact connection lives in TranscriptionService.swift in the Fazm Desktop sources:

// TranscriptionService.swift, lines 274-307
let url = "wss://api.deepgram.com/v1/listen"
  ?model=nova-3                  // line 94
  &language=en
  &encoding=linear16
  &sample_rate=16000
  &channels=2                    // mic (0) + system audio (1)
  &multichannel=true
  &interim_results=true
  &endpointing=300               // ms silence -> speech_final
  &utterance_end_ms=1000         // backup endpoint
  &vad_events=true
  &diarize=true
  &keyterm=<per-utterance term>  // app names, file names
  &replace=dot+com:.com  &replace=at+sign:@  ...

// Audio chunk size: 3200 bytes (100 ms at 16 kHz, 16-bit). Line 119.
// Keepalive: every 8 s, lines 348-374.
// Watchdog: every 30 s, reconnect after 60 s of stale data, lines 376-399.

The point of pasting it: this is the surface a local stack has to match before the cloud hop disappears. Not the model. The connection object around the model. The day FluidAudio or WhisperKit (or a peer) exposes those four query parameters as part of a stable streaming session API, the Fazm TranscriptionServicegains a local backend and the audio frame stops leaving the box. The architecture is already decoupled: the class hands frame-by-frame transcript segments to a closure, swap the WebSocket out for a local session emitting the same shape and the rest of the app does not notice.

The local voice stack on a Mac in May 2026 (what already ships)

For dictation specifically, this is solved. The chip row below is every shipping option we used or read the source of while writing this. All run on-device. None are voice agents in the full sense. That is the gap to fill.

What flips the answer to "fully yes"

Two paths to the local-everywhere voice agent loop on a Mac, ordered by how soon I think each lands.

Path A. An open-source on-device transducer (Parakeet TDT class, or a NeMo-family peer) exposes a streaming session API that covers runtime keyterm and multichannel-on-one-stream. FluidAudio or WhisperKit wires it up behind a Swift surface that an agent can target. The day that ships, every voice agent on a Mac that is willing to drop a cloud transcription dependency does so in one afternoon. The model is the part that is already strong enough.

Path B. Apple's macOS 26 SpeechAnalyzer ships in a point release with a public streaming API that covers per-utterance vocabulary boost and multichannel input. The first-token latency is already there. The licensing story is simpler than rolling a third-party model. The remaining unknown is which knobs the public API exposes versus which stay private.

Either path, the day it lands the audio frame stops leaving the machine and the answer to the question on the title becomes a flat yes. Until then, the honest answer is the one above: three layers out of four, with the fourth in motion.

If you can't wait for path A or B

Two reasonable shapes today.

Local dictation, cloud planning. Run Parakeet or WhisperKit on the Mac for the words. Send only the transcribed text plus a structured representation of the screen to your LLM provider. Audio never leaves the laptop. This is what most privacy-first dictation apps converge on. The tradeoff is the loop feels like dictation, not like voice control: no interim partials, no hard endpointing, no agent-driven barge-in.

Self-hosted streaming transducer. Run Deepgram on-prem, or stand up whisper-streaming or a faster-whisper server inside your network. Point the agent's transcription URL at that machine. The audio leaves the laptop but not your hardware. Closest existing path to a fully audited voice agent loop with the four streaming knobs intact.

Frequently asked questions