The four places an AX-tree chain breaks at app boundaries

Why the boundary is a structural problem, not a polish problem

The accessibility API is per-process. You build an AXUIElement from a PID, you walk its tree, you send AX messages to its elements. There is no “system tree” that contains every running app. There is one tree per process, and the tree for app A has no nodes belonging to app B in it.

A screenshot-driven agent has the opposite property: it sees pixels of whatever happens to be on top, with no notion of which process produced those pixels. Both surfaces have boundary problems, but they are different boundary problems. The screenshot agent never knows which app it is in. The AX-tree agent always knows which app it intends to act on, but if the world moves under it, the next call goes to a stale PID and the chain ends.

So the question is not “can the AX-tree path work across apps” (it can, the API supports targeting any running process), but “how does the agent learn the world moved.” That learning happens on the response of the action that caused the move. The four failure modes below are the four ways that response can communicate “the next call should not use the PID you just used.”

The dispatch shape

Every action call has the same shape: input is a PID plus an element target plus an optional inline chain (text, pressKey). Output is a diff of the original PID's tree, optionally plus a full traversal of a different PID that became frontmost as a side effect.

The interesting part is on the right side. A vanilla AX-tree binding would only return the first item: a diff of the app you targeted. The other two are how the boundary cases are surfaced without the model having to make an extra observation call to figure out what just happened.

The response struct, verbatim

Three fields carry the cross-app information. Two more carry the in-app modal information. The model decides which fields were populated and routes its next call accordingly.

When appSwitchPid is set, the next tool call uses that PID. When sheetDetected is set, the model knows the target window is occluded and should drive the sheet first. When neither is set, the chain stays in the original PID. That is the entire boundary contract.

Failure mode 1: a click in app A activates app B

The most common boundary case. The model clicks a link in Mail, which opens a browser tab. Or a button in a chat client, which opens a meeting app. Or anything that fires a deeplink via NSWorkspace.open under the hood. The click was honest, the target element existed, the AX action succeeded. The diff in the original app is small or empty. From the model's side, this looks like the click did nothing. From the user's side, an entire new app just opened.

The handler is a frontmost-PID comparison on the action response, attached for every diff-based tool call:

Three things to notice. First, the check runs unconditionally, not gated on a flag the model has to set. Second, the new app gets a full traversal in the same response, so the model never has to make a follow-up “what is this app I am in” call. Third, the original app's diff is still returned alongside the app-switch payload, because both are useful: the diff tells the model “the click landed and Mail closed the message”, the appSwitchTraversal tells it “you are now in Safari with this tree.”

Failure mode 2: the action itself is the app switch

Cmd+Tab is just a key press. macos-use_press_key_and_traverse accepts keyName: “Tab” plus modifierFlags: [“Command”]. The key event fires, the OS swaps the frontmost app, and the same frontmost-PID check from failure mode 1 picks up the change and traverses the new app. The handler does not care whether the switch was intentional. From the response shape it looks identical to a deeplink: appSwitchPid is set, and the next call uses that PID.

The same path works for clicking the Dock, clicking an item in the macOS app switcher overlay, and clicking a notification banner. All of them are AX actions whose side effect happens to be a process becoming frontmost. The server does not need a special “app switch tool”; the handoff falls out of the standard action plus diff plus frontmost-check pipeline.

Failure mode 3: an AXSheet covers the target window

Sheets do not cross the app boundary in the PID sense. The sheet still belongs to the same process as the parent window. But they create the same chain-breaking effect: the diff still contains all the buttons of the parent window, all of those buttons are in the AX tree, and in_viewport may report them as visible because their AX coordinates are inside the window frame. None of them are clickable, because the sheet is on top.

The handler walks the app's AXWindows looking for an AXSheet child (lines 243-278). When one is found, the in-app viewport gets re-scoped: viewport bounds are the sheet's frame, not the window's frame. Every diff element gets re-checked against the sheet bounds. Buttons behind the sheet flip to in_viewport: false, the sheet's buttons stay in_viewport: true, and the model's decision rule (“only target in-viewport elements”) does the right thing without any new logic.

This is the failure mode that bites people testing AX-driven agents in the wild. A Save dialog, a permissions prompt, a confirmation sheet: all of them break a naive “just-walk-the-AX-tree” agent because the tree looks fine and the model picks an unreachable target. Without the sheet-aware viewport, no amount of cross-app handoff detection helps, because no app switch happened.

Failure mode 4: a third app steals focus mid-action

A meeting starts. A Slack call comes in. A 1Password prompt appears. The agent was halfway through a click on the target app when a different process became frontmost. For screenshot-driven agents, the next screenshot shows the interrupting app and the chain pivots into nonsense. AX-driven actions have a structural defense: they are sent to a PID, not to whatever is on top. The click that already fired went to the original PID via AX API and does not care which window is drawn on top of it.

The server still has to clean up the visible state. Two mechanisms cooperate. First, the InputGuard engaged for every disruptive action (line 1835) blocks user input during the action and shows an overlay, so the human cannot accidentally click the interrupting app and confuse the agent further. Second, after the action, the frontmost-app restore reasserts the original frontmost so the user's work continues where they left it:

If the interrupting app is a real third process, the cross-app handoff detection from failure mode 1 still fires, and the model can decide whether the interruption is something to handle (answer the call) or something to defer (mark unread, return to the original target). The point is that the decision happens with full structured context, not on the basis of a screenshot of the new app drawn over the old one.

What the “just take a screenshot” path actually loses

A screenshot agent does not have process identity in its payload. There is no PID on a screenshot. So none of the four handlers above can be expressed in that surface. The closest analogue is a heuristic on the visual diff: did the menu bar change, did the app icon flash in the Dock, did the title bar rename. Those are guesses, and they fail silently when the interrupting app looks similar (a second instance of the same browser, a sheet rendered with the parent app's style, a modal from a helper process bundled into the main app).

The AX-tree path makes “which process am I driving” a structured field on every response. Screenshot pipelines treat it as a property of the camera angle. That is the gap. Everything else (latency, token cost, prompt size) is a consequence of the same structural difference.

Fazm consumes mcp-server-macos-use through its acp-bridge. The bridge does not implement boundary handling itself; it inherits all four behaviours from the MCP server, so any model wired through Fazm gets the same cross-app fields on every tool response without the bridge layer doing anything clever.