Accessibility Tree Desktop Automation: the text format an LLM actually reads

Every windowed app on macOS, Windows, and Linux exposes a hierarchical map of its UI to the operating system. Roots are the desktop and application windows. Children are menus, buttons, text fields, lists, and so on. Each node has a role (AXButton, AXMenuItem, AXTextField), a title, sometimes a value, and a frame in screen coordinates. VoiceOver, NVDA, and Orca read it to describe the screen out loud. A desktop automation tool can read it for the same reason: it is a structured, machine-friendly description of what is on screen, without any pixel parsing.

Three reasons. First, it is text, not pixels. A typical app's tree is a few kilobytes; a 4K screenshot is tens of megabytes uncompressed. Second, the LLM does not have to guess where 'the Send button' is by interpreting an image. The button is right there in the tree, with a role of AXButton, a title of 'Send', and a frame of x:842 y:1004 w:60 h:32. Third, the tree includes things pixels cannot: invisible items, role information, accessibility hints, and the exact bounding box. Pixels alone require a vision model to read back the UI, which is slower and more expensive on every step.

Fazm bundles a standalone Swift-native MCP server inside the app: Contents/MacOS/mcp-server-macos-use, version 1.6.0. The Node.js bridge that registers it lives at acp-bridge/src/index.ts, line 63 (binary path) and lines 1056 to 1064 (registration). The list of built-in MCP servers is hardcoded as fazm_tools, playwright, macos-use, whatsapp, google-workspace at line 1266. You can run the binary directly: pipe a JSON-RPC tools/list request into it and it will print the six tool names.

One line per AX element. The header shows the application name, element count, and traversal time, e.g. '# Fazm Dev — 441 elements (0.72s)'. Each element line is '[AXRole (subrole)] "text" x:N y:N w:W h:H visible'. A real example: '[AXMenuBarItem (menu bar item)] "Apple" x:1681 y:-2160 w:34 h:30'. Negative coordinates appear when you have multiple displays. The 'visible' flag at the end is set when the element is currently rendered, not just attached to the tree. The full file is saved to /tmp/macos-use/<timestamp>_<tool>.txt and a screenshot to a sibling .png.

All six tools end in _and_traverse: open_application_and_traverse, click_and_traverse, type_and_traverse, press_key_and_traverse, scroll_and_traverse, refresh_traversal. The suffix is not decorative. It means the tool performs the requested action and then immediately rebuilds the accessibility tree, returning both in a single response. The model never has to call a separate 'see what changed' tool after a click. Round-trip cost goes from two MCP calls to one, and the tree the model reasons against is always less than a second old.

The mcp-server-macos-use binary is macOS-only because it talks to AXUIElement, the native macOS accessibility C API. The same architectural pattern (read accessibility tree, send compact text to model, post action back to OS) works on Windows via UIAutomation and Linux via AT-SPI. Fazm's Mediar group also ships an MIT-licensed Rust library called Terminator that wraps both UIAutomation on Windows and AX on macOS, which is what Fazm uses for cross-platform automation features. None of that requires the user to install a separate runtime: on macOS, the binary is bundled inside Fazm.app.

No. Fazm is a consumer Mac app, not a developer framework. You install it, grant Accessibility permission once (the same TCC dialog VoiceOver uses), and the macos-use MCP server is registered automatically alongside Playwright, WhatsApp, and Google Workspace. You then talk to Fazm in plain English. Behind the scenes, the model picks tools, the bridge routes them, and the accessibility tree text comes back into the LLM's context. No Python, no Node, no manual MCP config. Free to start.

Anthropic's reference Computer Use implementation and OpenAI Operator are both pixel-first: they capture the screen, send it to a vision model, and click on coordinates the model returns. That is general but slow, expensive, and blind to non-visible structure. Accessibility-tree automation reads structured text, which is faster and lets you click 'Send' by name rather than by image patch. The trade-off is that some apps (Qt, OpenGL games, certain Electron apps) implement the AX protocol poorly, and you fall back to pixels. Fazm uses both: AX first, vision when AX is empty.