Why Typed Tools Matter for Desktop Automation Agents
Why Typed Tools Matter for Desktop Automation Agents
The conversation about backend infrastructure for coding agents misses the bigger picture. The typed tools approach - giving agents structured, schema-validated interfaces instead of raw API access - extends way beyond backend infra. Desktop automation hits the exact same wall, and the macOS accessibility API is the perfect example.
The Accessibility API Problem
The macOS accessibility API is basically a loosely structured tree. You query it and get back a hierarchy of UI elements with attributes like role, title, value, and children. But there is no schema. A button might have a title or it might not. A text field might report its value through the value attribute or through a child static text element.
Without typed tools wrapping this API, an AI agent has to guess at the structure every time. It queries the tree, interprets what it finds, and hopes the pattern holds for the next app. It usually does not.
What Typed Tools Fix
When you wrap the accessibility API in typed tool definitions, you give the agent a contract. Instead of "query this tree and figure it out," the agent gets structured actions like:
- clickButton(appName, buttonTitle) - finds and clicks a button by its accessibility label
- readTextField(appName, fieldIdentifier) - extracts the current value from a text input
- listMenuItems(appName, menuName) - returns all items in a specific menu
Each tool has defined inputs, expected outputs, and error cases. The agent does not need to understand the raw accessibility tree - it works with clean abstractions.
Beyond Desktop
This pattern applies everywhere agents interact with loosely structured systems - web scraping, file system operations, system configuration. Any time the underlying API returns unstructured or semi-structured data, a typed tool layer makes the agent dramatically more reliable.
The lesson from backend infrastructure is universal: agents need structure, not raw access.
Fazm is an open source macOS AI agent. Open source on GitHub.