One chain link is one action plus an AX-tree diff

One tool call that does two things in a single response: it performs an action (click, type, press a key, scroll, set a value, set selected, press an AX action) on a target Mac app, and it returns the accessibility-tree diff between the moment before the action and the moment after. The diff has three buckets: added nodes, removed nodes, and modified nodes (with attribute-level before/after deltas). The model reads the diff and picks the next action. There is no separate 'see what changed' tool call. The action and the observation are fused into one round-trip.

A real macOS app's AX tree is hundreds of nodes deep. Returning the full tree after every click means the model spends most of its context reading mostly-unchanged scaffolding. The diff is two orders of magnitude smaller: a typical click that opens a menu adds maybe 8 to 20 nodes, modifies 1 to 3, and removes nothing. The model reads exactly the part of the page that changed because of the action it just took, which is what it needs to decide the next action. Full traversal is reserved for the two cases where it is actually useful: opening an app for the first time, and explicit `refresh_traversal` calls when something looks off.

Three classes of noise. (1) Pure coordinate moves: any modified node whose only changed attributes are x, y, width, or height is dropped (`coordinateAttrs: Set<String> = ["x", "y", "width", "height"]`, line 649 of mcp-server-macos-use/Sources/MCPServer/main.swift). Layout reflows after a window resize would otherwise flood the diff. (2) Scroll-bar churn: scrollbar, scroll bar, value indicator, page button, and arrow button roles are dropped via `isScrollBarNoise` (lines 592-597). Scrolling triggers tons of these, none of them are actionable. (3) Empty structural rows, cells, columns, and menu containers without text are dropped via `isStructuralNoise` (lines 600-607). The result is a diff the model can actually reason against in 100-300 tokens.

Every element in the diff carries `in_viewport: Bool?`. It is set by checking whether the element's center point falls inside any of the app's window rectangles (or the bounds of an active AXSheet, when one is open). The accessibility tree includes elements that exist in the tree but are scrolled off-screen or in a hidden tab. Without the viewport check, the model often clicks an element it found in the tree, the click coordinates land outside the visible window, the OS routes it to whatever is on top there, and the chain is dead. With the flag, the model knows which added elements are actually visible and clickable right now, and falls back to a scroll if the target is in_viewport=false.

click_and_traverse accepts optional `text` and `pressKey` parameters (lines 1678-1684 of MCPServer/main.swift). If both are set, the server clicks the target, types the text, presses the key, and only then runs the post-action traversal. That is one model turn, one tool call, one diff. Without this, sending a Slack message would be three separate tool calls (click into the input, type the message, press Return) and three separate diffs to read. type_and_traverse has the same trick with its own `pressKey` argument (lines 1694-1698). The MCP server's instructions explicitly tell agents to chain inside a single call: 'do NOT split into separate click, type, and press calls.'

Cross-app handoff is detected and surfaced in the same response. The ToolResponse struct carries optional `appSwitchPid`, `appSwitchAppName`, and `appSwitchTraversal` fields (lines 227-230). After the action, the server checks the current frontmost app and if it changed, it traverses that app too and includes its tree alongside the diff for the original target. The model sees both 'here is what changed in the app you clicked into' and 'oh, also you are now in this other app, here is its tree.' That makes Cmd+Tab, Open With, deep-linking out of an email, and any other app-launching action a single chain link instead of two.

Because the AX tree contains elements that exist but are not currently scrolled into the visible region of the window. If the model picks coordinates from such an element and the click fires at those raw coordinates, the click lands somewhere else (or off-screen, depending on the OS). Before clicking, the server calls `scrollIntoViewIfNeeded` (line 1662) which checks the target point against window bounds and scrolls the relevant container so the point becomes visible, then translates the click coordinate to wherever the element ended up. The model writes 'click element X' once; the server handles the visibility plumbing.

A screenshot-driven chain is 'take screenshot, model picks pixel coords, click, take screenshot, model picks pixel coords, click.' The model has no symbolic identity for any element. The 'diff' between two screenshots is a pixel diff, which is information-poor and noisy: a flashing cursor, an antialiasing change, a progress spinner all show up as differences with no semantic meaning. The AX-tree diff is symbolic: the model gets the role, the text, the attribute changes (like AXValue going from empty to 'hello world'), and the visibility flag, all without sending a single image. Round-trip is faster, the prompt is dramatically smaller, and the chain is reproducible because role+text identifies a button across app updates while pixel coordinates do not.

It is open source. The structs and the filtering live in mcp-server-macos-use/Sources/MCPServer/main.swift in the public repo. EnrichedTraversalDiff is at lines 212-216, DiffElementData at lines 187-196, the filter pipeline at lines 648-718, the noise predicates at lines 592-607, and the per-tool action handlers at lines 1598-1795. The MCP server version on the binary is 1.6.0 (line 1487). Fazm consumes this server through its acp-bridge, but the chain mechanics described here are entirely the MCP server's job; the bridge sits a layer up.

For native apps the AX-tree action chain is the primary path. For the browser, Fazm uses a Playwright bridge that exposes the rendered DOM through the same accessibility surface (the browser's own AX tree, not the raw DOM), which gives a chain shape that is similar but not identical: the diff is computed against accessibility-tree snapshots of the page rather than against the OS-level macOS AX tree. The principle, action plus diff plus visibility flag, is the same in both. The mechanics of how diffs are computed and noise filtered are tuned per surface.