Switching from DOM Selectors to Accessibility Tree Cut Our Flake Rate from 30% to 5%
Switching from DOM Selectors to Accessibility Tree Cut Our Flake Rate from 30% to 5%
We ran browser automation across 100 daily interactions for three months. With DOM selectors, our flake rate sat at 30%. After switching to the accessibility tree, it dropped to 5%. Here is why.
Why DOM Selectors Are Fragile
DOM selectors target elements by their HTML structure - class names, IDs, XPath expressions, CSS selectors. The problem is that web developers change these constantly. A framework update renames CSS classes. A redesign restructures the div hierarchy. An A/B test swaps component layouts. Your selector worked yesterday and fails today, even though the button is still right there on the page.
With 100 daily interactions across multiple websites, at least one site changes something every day. Each change potentially breaks multiple selectors. Maintaining a selector-based automation system means constantly playing catch-up with website updates.
Why the Accessibility Tree Is Stable
The accessibility tree represents semantic meaning - "this is a Submit button," "this is a search text field," "this is a navigation menu." These meanings rarely change even when the underlying HTML is completely restructured.
A website can swap from React to Vue, change every CSS class, and restructure the entire DOM hierarchy. As long as there is still a Submit button on the page, the accessibility tree still has a Submit button. The label might change ("Submit" to "Send"), but the role (button) and the semantic purpose stay the same.
The Remaining 5%
The flakes that remain after switching to the accessibility tree fall into three categories. Dynamic content that has not loaded yet - the element exists in the DOM but is not yet in the accessibility tree. Custom widgets that do not properly expose accessibility properties. And legitimate UI changes where a button is removed or renamed.
These are solvable with waits, fallbacks, and periodic re-mapping. The 25-percentage-point improvement from just switching the perception layer was the easy win. Getting from 5% to under 1% requires more investment but is achievable.
Fazm is an open source macOS AI agent. Open source on GitHub.