Mac launcher AI agent accessibility, the part nobody else wires up

What the launcher is, and what it is not

The Mac launcher genre has had one job for twenty years: type a few characters, hit enter, an app opens. Spotlight added file results and definitions, Alfred added workflows, Raycast added extensions. The consistent shape is a global hotkey that drops a fuzzy-matched picker in front of you and returns control to whatever you were doing.

The chat-bar versions of the same UI (Raycast AI, MacGPT, the various menubar wrappers) added a second job: type a sentence, hit enter, an LLM replies with text. The hotkey is the same; the result is text in a box. That is a useful surface for definitions, drafting, and quick translations. It is not what this post is about.

The thing this post is about is the third generation: the launcher where enter does not call a chat completion, it calls an agent loop. The agent reaches into whatever app was in front of you a moment ago, reads its accessibility tree, and clicks, types, or scrolls there. Voice input is the same surface with a microphone. The launcher is still the entry point. The accessibility API is what the launcher is standing on top of.

One input, many surfaces

The launcher does not care which app is in front of you. The agent picks the right reach based on the frontmost window: a native AX tree for Mail or Slack, a browser-extension path for Chrome page DOM, a Google Workspace tool for Docs/Sheets/Calendar, a screenshot fallback for the Qt or OpenGL apps that do not expose a tree at all.

The single hub in the middle is the part that makes this design hold together. The launcher does not branch into N different code paths based on what app you have in front of you. It sends one prompt to one agent, and the agent calls the right tool based on what it sees. The AX tree is the default, because it is two orders of magnitude faster than vision, has labels for free, and never needs OCR.

The hotkey wiring, in the actual code

This is the bit that surprised me when I started writing it. The launcher uses Carbon's RegisterEventHotKey, not the newer NSEvent.addGlobalMonitor path. The Carbon hot-key API has been deprecated-but-still-supported for fifteen years and is what Alfred, Raycast, and most launchers actually use under the hood. The reason: it works without the Accessibility TCC permission, so the launcher can open before the first permission prompt ever fires.

Three hotkeys get registered. Cmd + \ (keycode 42) is the toggle for the floating bar. The Ask shortcut is user-configurable: Cmd Enter, Cmd Shift Enter, Cmd J, or Cmd O depending on what the user has set in Settings. The third pops a new chat window into a standalone window. The FourCharCode signature passed to EventHotKeyID is 0x46415A4D, which is the literal bytes for ASCII "FAZM".

The handler on the other end of those registrations posts a DistributedNotification. The bar shows up, focused. If the user picked the Ask shortcut, the cursor lands in the agent input field; if they picked the toggle, the bar shows whatever conversation they last had. A separate observer on ShortcutSettings.askFazmShortcutChanged re-registers the Ask hotkey live, so changing it in Settings does not need a restart.

What the agent calls, once the bar opens

The launcher hands the prompt to the agent loop over ACP (Agent Client Protocol). The bundled agent is Claude Code by default, with Codex available as a swappable backend per chat. Both run with one extra MCP server: a native macOS accessibility automation server bundled inside the app at Contents/MacOS/mcp-server-macos-use. The bridge code that discovers and starts it is at acp-bridge/src/index.ts, line 100, where it resolves the path; the server gets registered in buildMcpServers around line 2070 under the name macos-use.

The tools the agent sees are named things like macos-use_open_application_and_traverse, macos-use_click_and_traverse, macos-use_type_and_traverse, macos-use_press_key_and_traverse. Each one performs the action and returns a flat text dump of the resulting AX tree, line per element, format [Role] "text" x:N y:N w:W h:H visible, plus a screenshot path on disk that the agent can read separately if it needs to disambiguate.

The chain reaches into the AX C API: AXUIElementCreateApplication(pid) to get a root handle on a target process, AXUIElementCopyAttributeValue with kAXChildrenAttribute, kAXRoleAttribute, kAXTitleAttribute, kAXValueAttribute to walk the tree, AXUIElementPerformAction(kAXPressAction) to click, CGEventCreateKeyboardEvent to type. No pixel sampling unless the AX tree is empty.

The permission probe (because cannotComplete lies)

A launcher that drives the AX tree has to handle one quirk that the documentation does not warn you about: the error kAXErrorCannotComplete is ambiguous. It can mean the system has revoked accessibility permission, or it can mean the frontmost app simply does not expose an AX tree at all. Qt apps, Python apps that draw to a single Cocoa view, OpenGL canvases, and a few Electron builds with accessibility disabled all return the same error code. If the launcher reads it as "permission broken" every time, it shows the Settings nag every time the user happens to have PyMOL in front.

The probe lives at lines 482 to 534 of Desktop/Sources/AppState.swift. If the AX call against the frontmost app returns cannotComplete, the probe retries against Finder, which is canonical and always exposes a tree. If Finder also fails, the permission is actually broken and the launcher prompts the user to open System Settings. If Finder succeeds, the launcher knows the frontmost app is at fault and quietly falls back to coordinates plus a screenshot. The launcher does not lie about permission state to make its own error path easier.

Why the accessibility path, not screenshots

A screenshot agent and an AX agent solve the same problem with two very different cost profiles. A full BFS walk of a focused window in Slack, Mail, or Chrome runs in 30 to 80 milliseconds end to end, measured on the same machine that is running the launcher. A screenshot pipeline takes 200 to 400 ms to capture, sends a 1 to 5 MB PNG to a vision model, waits 1 to 4 seconds for inference, and parses pixel coordinates back. The AX path is roughly two orders of magnitude faster on a typical action.

The cost split also changes what travels off the machine. A screenshot agent sends pixels for every action: the contents of your inbox, the message you are about to reply to, the document on screen. An AX agent sends labels and roles for the elements it actually touches: the title of the button, the text of the field, the value of the focused control. A launcher you invoke fifteen times a day with a screenshot pipeline produces a lot more leakage than the same launcher with an AX pipeline.

The flip side: the AX tree is empty for the apps that need a vision pass the most (creative tools, scientific software, custom canvases). A serious launcher does both. AX first because it is fast and cheap; screenshot fallback because some surfaces have no tree to read. Defaulting to vision is the lazy choice. Defaulting to AX is the design choice.

Voice input is the same launcher

The voice path is not a separate product surface. It is the Ask shortcut, with a microphone capture in front of the text path. Hold the key, talk, release; transcription runs on-device through WhisperKit, the text version reaches the same agent loop, the same macos-use tools resolve against the same frontmost-app pivot.

The reason this works at all is that the agent is the executor. "Reply to that one with a Tuesday slot" is not a fixed intent matched against a skill list; it is a sentence the model reads and resolves against the AX tree of whatever Mail thread is in front of you. The launcher does not need to know what "that one" is. The model and the AX tree do.

The three things the launcher itself never does

A clean separation between the launcher and the agent makes the surface smaller and easier to reason about. The launcher specifically does not:

• Call any model. Hitting enter posts a notification with the prompt; the agent loop in a separate process is what handles inference.
• Read the screen. The launcher does not own a screenshot path. The macos-use MCP does, and only when the agent decides to call it.
• Track context. Persistent session state lives in the agent layer, with an explicit chain of upstream session IDs. The launcher is stateless between keystrokes.

Keeping those out of the launcher is what lets the launcher feel instant. A 4 millisecond Carbon hot-key handler is the only thing between Cmd+\ and a visible input field. Everything else happens after the user types.