Accessibility APIs Are the Cheat Code for Desktop AI Agents
Accessibility APIs Are the Cheat Code for Desktop AI Agents
Every desktop AI agent needs to answer the same question: what is on the screen right now? Most agents take screenshots and send them to a vision model. This works but it is slow, expensive, and imprecise. There is a better way that has existed on macOS since 2001.
AXUIElement - Semantic Screen Understanding
macOS exposes every application's UI through the Accessibility framework. AXUIElement gives you the complete tree of interface elements - buttons, text fields, labels, menus, checkboxes - with their exact positions, roles, values, and states.
This is not pixel-level guessing. When you query AXUIElement, you get structured data: "There is a button labeled 'Send' at position (450, 320), it is enabled, and it is inside a toolbar." You know what it is, where it is, and whether you can interact with it.
Why This Beats Screenshots
A screenshot sent to a vision model costs tokens, adds latency, and produces approximate results. The model might misread text, confuse similar-looking buttons, or miss elements entirely. AXUIElement returns exact data in milliseconds with zero API cost.
For desktop agents, this means faster action loops, more reliable element targeting, and dramatically lower operating costs. An agent using accessibility APIs can check screen state 10 times per second. An agent using screenshots might manage once every 2-3 seconds.
The Catch
Not every application exposes a complete accessibility tree. Electron apps generally do. Native macOS apps almost always do. Games and custom-rendered canvases do not. For those edge cases, you fall back to screenshots.
The practical approach is accessibility-first with screenshot fallback. Use AXUIElement for everything it covers - which is the vast majority of productivity software - and only invoke the vision model when the accessibility tree is empty or incomplete.
This is not a new technology. Screen readers have used it for decades. Desktop AI agents are just the latest beneficiary of infrastructure built for a completely different purpose.
Fazm is an open source macOS AI agent. Open source on GitHub.