A voice controlled macOS agent that actually clicks buttons in Slack, Linear, and Notion.

A voice controlled macOS agent is software that takes a spoken instruction, understands it with a language model, and then performs the resulting actions against your Mac's real applications, not just a text field. The category splits into four types. Pure dictation apps (Spokenly, Superwhisper, Wispr Flow) stop at transcribed text into whatever window is focused. Apple's built-in Voice Control is a literal command grammar, not an agent; it cannot interpret intent like 'mark the top three threads in Slack as read.' AppleScript-based agents (GPT-Automator) die on Electron apps like Slack, Linear, Notion, Cursor, Figma. Screenshot-vision agents (OpenInterpreter, Anthropic Computer Use in screenshot mode) spend 500ms to 2s per step on vision inference and occasionally click the wrong pixel. Fazm is in a fifth bucket: voice plus a language model plus native macOS accessibility APIs.

Five stages. (1) Audio is captured by CoreAudio HAL IOProc at 16 kHz mono in AudioCaptureService.swift, avoiding AVAudioEngine's aggregate-device trick that degrades Bluetooth output quality. (2) Push-to-talk is gated on the Option or Left Control modifier key in PushToTalkManager.swift, with a 400 ms double-tap threshold to switch between hold-to-talk and locked-listening. (3) Audio chunks stream to Deepgram nova-3 over a WebSocket at 100 ms cadence; interim results appear within ~200 ms. (4) The transcript lands in the same ACP bridge that Fazm's chat uses, and the model picks a tool call. (5) When the task is a macOS app action, the model emits an mcp__macos-use__* call, which is handled by the bundled Rust binary mcp-server-macos-use at Contents/MacOS/mcp-server-macos-use. That binary walks the AXUIElement tree of the target app and returns a text file where each element is `[Role] "text" x:N y:N w:W h:H visible`, then clicks at (x+w/2, y+h/2).

Three reasons. Latency: a full AX traversal of a running app is sub-100 ms; a vision round-trip through a VLM is 500 ms to 2 s per step. Ten steps in a workflow is the difference between one second and twenty. Precision: AXUIElement returns exact coordinates the OS itself uses for hit-testing, which means the click lands on the real element regardless of DPI, Retina scaling, dark mode, or theme. Screenshot-vision can click the wrong pixel when icons look similar or when a modal half-covers a target. Off-screen access: the AX tree includes elements that are scrolled out of view. A screenshot doesn't. On the other hand, the AX tree is empty on canvas-based apps like Figma or games, which is where screenshot-vision is still the right choice. Fazm uses AX first and screenshots as a targeted fallback.

Two files tell the whole story. `/Users/matthewdi/fazm/acp-bridge/src/index.ts` line 63 sets `const macosUseBinary = join(contentsDir, "MacOS", "mcp-server-macos-use");` and line 1059 registers it as the `macos-use` MCP server alongside the browser runtime. The binary itself is shipped inside the signed and notarized .app bundle, so `codesign -dv /Applications/Fazm.app/Contents/MacOS/mcp-server-macos-use` returns the same developer ID as Fazm itself. The MCP server returns its traversal as a text file saved under /tmp/macos-use/, one line per element, format `[Role] "text" x:N y:N w:W h:H visible`. Clone the public repo and run `rg -n mcp-server-macos-use acp-bridge/src/index.ts` to see the registration line for yourself.

`Desktop/Sources/TranscriptionService.swift:94` sets `private let model = "nova-3"`. Line 97 sets `private let sampleRate = 16000`. The WebSocket connects to Deepgram with `interim_results=true`, `diarize=true`, smart formatting on, and a 500-token custom vocabulary you can tune for the apps and names you use often. Audio frames are 3200 bytes each (~100 ms) and are flushed immediately via `no_delay=true`. On the input side, `AudioCaptureService.swift` installs a CoreAudio IOProc directly on the selected input device, preferring physical mics in Built-in > USB > Bluetooth order and avoiding virtual/aggregate devices. On the trigger side, `PushToTalkManager.swift` runs a four-state machine (idle → listening → lockedListening → finalizing) keyed off Option, Left Control, or Fn, with a 0.4 s double-tap threshold and a 5-minute safety cap.

Yes, and this is the entire reason the approach matters. Electron apps are Chromium in a native shell, and Chromium exposes a rich accessibility tree by default. A voice command like 'mark the top three threads in Slack as read' walks the Slack AX tree, finds the thread rows by role and label, and clicks each 'Mark as read' affordance by coordinate. Linear, Notion, Cursor, VS Code, Discord, and 1Password all behave the same way. AppleScript is dead in these apps; a standard AppleScript 'tell application Slack' call cannot reach a message thread. The AX approach works because Chromium bridges the DOM to NSAccessibility. This is the same reason the macos-use MCP server works on native apps like Finder, Mail, Settings, Numbers, Messages, and Reminders.

Canvas-based applications like Figma, raw Photoshop canvases, and most games. These apps draw their UI without exposing elements to NSAccessibility, so the AX tree for the document area is effectively empty. For those, the agent has to fall back to screenshot-vision. Remote desktops and VMs also lack a local AX tree for the remote UI; a Citrix or Parsec session is pixels from Fazm's perspective. Fullscreen video and browser-embedded games are the same. Practically, this means Fazm is strongest on the exact apps knowledge workers spend most of their day in (chat, docs, mail, CRM, Linear, IDEs, Settings, Finder) and weakest on creative canvas work. For the latter, a screenshot-vision tool is the better fit.

The PTT manager supports two usage patterns so voice is usable beyond a 5-second demo. Hold-to-talk means you hold Option while you speak and it stops the moment you release it; the 5-minute safety cap in PushToTalkManager.swift (line 65, `maxPTTDuration: TimeInterval = 300`) prevents a stuck modifier from recording forever. Double-tap-to-lock means you tap Option twice within 0.4 s and it enters lockedListening; a third tap finalizes. This is the mode for 'dictate this email' or 'keep transcribing while I think out loud.' The Left Control key has a delayed activation (controlDelayWorkItem) so Ctrl+C and Ctrl+V don't trigger a voice session accidentally. Audio input avoids AVAudioEngine's implicit aggregate device creation, which means switching between Bluetooth and Built-in mic mid-day does not audibly degrade system audio output.

Deepgram's nova-3 runs in the cloud. The audio stream leaves your Mac over a WebSocket to Deepgram's API. This tradeoff buys two things: quality (Nova-3 matches Whisper-large-v3 on general speech and is noticeably better on code, company names, and tools vocab via the 500-token custom dictionary) and low latency (partial results in roughly 200 ms against the 600 to 1200 ms typical for local Whisper.cpp on an M1 Air). If you want fully-local transcription, the source is open (github.com/mediar-ai/fazm) and the transcription layer is swappable; a community branch has already wired Whisper.cpp as a drop-in. For most users, the Deepgram path is strictly faster and more accurate, which matters more than local purity when you are talking to your Mac all day.

Apple Voice Control is a rule-based grammar on top of the AX tree; it knows how to do 'click Submit' but cannot interpret 'approve the second pending PR' because it has no language model. Siri is conversational but cannot actually drive arbitrary third-party Mac UIs; its action surface is the SiriKit intents Apple approves. Talon Voice is powerful and fully local, but it is a developer tool; you write grammars in Python, and it does not reason about intent the way an LLM does. Fazm is the consumer-friendly point in the space: a signed, notarized, auto-updating .app with an LLM deciding what to do and native AX APIs doing it. You install it, grant Accessibility and Microphone permission, hold Option, and talk.