Bracket Is a Speculation Play: Bet on Accessibility APIs

Fazm Team··2 min read

Bracket Is a Speculation Play

In desktop automation, there are two approaches: screenshot-based (look at pixels, infer UI state) and accessibility API-based (read the structured UI tree directly). The bet on accessibility APIs is a speculation play, and it is paying off.

40% to 90%

A year ago, accessibility API-based automation was unreliable. Coverage was spotty. Many apps did not expose their UI elements properly. Success rates for complex workflows hovered around 40%.

Today, the same approach hits 90% reliability. What changed was not the technology - the accessibility APIs have been stable for years. What changed was the ecosystem. More apps adopted proper accessibility support, driven by legal requirements and platform pressure. Apple tightened its accessibility guidelines. SwiftUI apps get accessibility support almost for free.

Why Screenshots Plateaued

Screenshot-based approaches hit a ceiling around 70-75% reliability. The problem is fundamental: you are trying to extract structured information from unstructured pixels. OCR errors, resolution changes, dark mode, custom themes, overlapping windows - every visual variation is a potential failure point.

More compute and better vision models help, but they cannot close the gap entirely. The information loss from rendering UI to pixels and then trying to recover the structure is inherent.

The Bet

Accessibility APIs give you the structure directly. No information loss. The button is a button, with a label, a position, and an action. You do not need to figure out what is a button by looking at pixels.

The speculation is that accessibility API coverage will keep improving because the incentives are aligned - accessibility benefits disabled users, automation, and testing. Everyone wants better accessibility support. Nobody is investing in making apps harder to automate via screenshots.

Bet on the approach where the trend is your friend.

Fazm is an open source macOS AI agent. Open source on GitHub.

More on This Topic

Related Posts