Desktop AI agent, beyond demos: the macOS edge cases a shipping Mac agent has to ship code for

Demos run on a machine that was set up five minutes ago with fresh TCC permissions, a single reference app, and no macOS update in flight. Real installs live through OS updates, app re-signs, background apps that expose partial or broken accessibility trees, and network tool calls that hang. The demo version of a desktop agent does not need to handle AXError.cannotComplete, does not need to disambiguate stale AXIsProcessTrusted() responses, and does not need per-tool-class timeouts. The shipped version does. That is the entire gap between a compelling screen recording and a product that a non-technical user can leave running unattended.

It caches the TCC (Transparency, Consent, and Control) decision on the first call and does not always invalidate that cache when macOS updates, when the app is re-signed, or when the user toggles the permission in System Settings. On macOS 26 Tahoe the per-process cache is particularly sticky. Fazm's comment at AppState.swift:308 calls it out explicitly: 'AXIsProcessTrusted() can return stale data after macOS updates or app re-signs'. The symptom is that the agent silently stops controlling apps even though System Settings shows the permission as granted. A naive agent just freezes. Fazm disambiguates using two independent signals (a Finder AX call and a CGEvent tap) before deciding the permission is really broken.

confirmAccessibilityBrokenViaFinder at AppState.swift:468 takes a running Finder process, creates an AXUIElement for it, and asks for its focused window. Finder is a known-good AX-compliant app that Apple ships with the OS, so a call against it should always succeed when accessibility permission is truly granted. The probe exists because AXError.cannotComplete from an arbitrary app can mean two very different things: either permission is actually broken, or that particular app's accessibility tree is just non-compliant or temporarily unresponsive. If Finder also fails, Fazm logs 'AXError.cannotComplete confirmed by Finder, permission is truly stuck' and enters recovery. If Finder succeeds, Fazm treats the original error as app-specific and keeps running.

probeAccessibilityViaEventTap at AppState.swift:490 is the tie-breaker. It calls CGEvent.tapCreate with tap: .cgSessionEventTap and options: .listenOnly, listening for mouseMoved events. Event tap creation is special because it queries the live TCC database directly rather than the per-process cache that AXIsProcessTrusted() uses. If the tap is created successfully, permission is real, even if AXIsProcessTrusted() currently says otherwise. The tap is invalidated immediately after the check so it does not consume system resources. That two-signal design is how Fazm survives the stale-cache case on macOS 26 Tahoe where AXIsProcessTrusted() continues returning false for minutes after the user grants the permission in System Settings.

It runs a tool timeout watchdog in the ACP bridge, at acp-bridge/src/index.ts lines 72 to 101. Three tiers: TOOL_TIMEOUT_INTERNAL_MS = 10_000 for internal tools like ToolSearch, TOOL_TIMEOUT_MCP_MS = 120_000 for any tool whose name starts with 'mcp__' (macos-use, whatsapp, google-workspace, playwright, and any user-added servers), and TOOL_TIMEOUT_DEFAULT_MS = 300_000 for the long tail. The watchdog tracks every running tool in an activeToolTimers map, and when a tool exceeds its tier's wall clock, it synthesises a completed-with-error frame so the agent loop can recover and the Swift bridge unblocks. Users can override the whole thing via the FAZM_TOOL_TIMEOUT_SECONDS environment variable from Settings, Advanced, Tool Timeout.

Primary input is the macOS accessibility tree via AXUIElementCreateApplication, AXUIElementCopyAttributeValue, and the AX property chain. Screenshots are used as a secondary signal for visual context when needed, captured synchronously through CGWindowListCreateImage in ScreenCaptureManager.swift and filtered by PID and window size. This is the opposite of screenshot-first agents like Anthropic Computer Use, which takes a picture, asks the model to read it, and hopes it clicks the right pixel. Accessibility APIs are fast (hundreds of milliseconds per action versus 2-5 seconds per screenshot), structured (the model gets a tree of roles and labels, not pixels), and model-neutral (no dependence on vision benchmark scores).

On a normal action, Swift-side AppState owns the accessibility permission health state (polled every 5 seconds, up to 3 retries), and the ACP bridge owns tool invocation. The agent loop selects a tool (say macos-use_click_and_traverse), the bridge starts a TOOL_TIMEOUT_MCP_MS timer, forwards the request to the mcp-server-macos-use binary, waits for its result via a named pipe (FAZM_BRIDGE_PIPE), and either cancels the timer on success or fires the synthetic completion frame on timeout. On the Swift side, the action may call AXUIElementCopyAttributeValue which returns .cannotComplete, which triggers confirmAccessibilityBrokenViaFinder, which either keeps the loop alive or flips isAccessibilityBroken to true and prompts recovery. The interesting part is that both the bridge timeout and the AX probe operate in the background: the user does not see them unless something fails.

Because tool classes have fundamentally different response profiles. ToolSearch and similar internal tools execute locally and should return in under a second, so a 10-second cap is loose enough to tolerate disk latency but tight enough to surface a broken tool quickly. MCP tools hit external processes (a browser, a native app, a Python server) and can legitimately take tens of seconds, so 120 seconds is the pragmatic ceiling. The default 300-second bucket is for anything that might legitimately need a long session, like a multi-step refactor across a large codebase. Collapsing all of these into one global timeout means either the fast tools hang too long on error or the slow tools get killed mid-session. The three-tier split is a small amount of code that absorbs a large amount of real-world variance.

Yes, via the FAZM_TOOL_TIMEOUT_SECONDS environment variable, read at acp-bridge/src/index.ts:82. Settings, Advanced, Tool Timeout surfaces the same value in the UI. When set to a positive integer, it overrides all three tiers with one uniform value in seconds. The design goal is that power users can dial it up for pathological workloads (a large sync job, a stubborn browser session) while the default tiers remain tight enough for the common case. Setting it to zero returns the tiered behavior. This is the same pattern as the reentrant warmup delay and several other knobs: sensible default, simple env override, no config file required.

Any macOS app that exposes an accessibility tree works out of the box because Fazm operates on the AX property chain, not on per-app adapters. The macos-use MCP server exposes generic click_and_traverse, type_and_traverse, scroll_and_traverse, press_key_and_traverse, and refresh_traversal tools that walk any AX tree. The pre-built MCP servers (whatsapp, google-workspace, playwright) exist only because those surfaces are either Catalyst apps with imperfect AX trees (WhatsApp) or non-macOS surfaces entirely (Google Workspace APIs, Chromium automation). For anything else, the generic macos-use flow handles it. That is why you can use Fazm with a random menu-bar utility, an old pre-notarized tool, or your company's internal Electron app, without writing an integration.

Yes, the Fazm desktop source tree is public. The consumer-facing product ships as a free Mac app with a paid tier, and the engine is buildable from source. The accessibility-permission probe code at AppState.swift:308-504 and the tool-timeout watchdog at acp-bridge/src/index.ts:72-101 are both in the public tree. If you want to verify the claims in this guide, those are the two files to read. That is a deliberate stance: the parts of a desktop agent that survive real installs are exactly the parts that benefit most from public scrutiny, because they are the parts you cannot fake in a screen recording.

Permission recovery code. It is boring, invisible, and easy to skip in a demo, but it is the largest source of user-visible failures in a shipped product. Every other desktop agent roundup talks about model choice, tool taxonomy, or token cost. Almost none of them talk about what the agent does when AXIsProcessTrusted() starts lying on a Tuesday afternoon after a macOS update ships. Fazm's answer is the Finder fallback probe plus the CGEvent tap probe plus a 5-second polling timer with up to 3 retries. That is three small blocks of code that absorb an enormous amount of user pain. The demo-to-reality gap in desktop agents is mostly code like that.