Beyond Apple Music MCP - Using Accessibility APIs to Control Any macOS App
Beyond Apple Music MCP - Controlling Any App with Accessibility APIs
App-specific MCP servers are a great start. An Apple Music MCP lets Claude search your library, play songs, and manage playlists. But building a separate MCP server for every macOS app does not scale. There is a better approach.
The Per-App MCP Problem
Every native macOS app has its own AppleScript dictionary, its own URL scheme, and its own quirks. Building an MCP server for Apple Music is one project. Building another for Finder is another. Safari, Mail, Calendar, Notes - each one is a separate integration effort.
And that only covers Apple's own apps. Third-party apps like Figma, Slack, or Arc have their own automation interfaces, if they have them at all. The per-app approach means you can only automate apps someone has built an MCP server for.
Accessibility APIs as Universal Control
The macOS accessibility API provides a universal interface to every application. It exposes the UI element tree - buttons, text fields, menus, windows - with semantic labels and roles. An MCP server built on top of this API can interact with any app, not just ones with dedicated integrations.
Instead of calling apple_music.play_song("title"), the accessibility-based approach finds the Apple Music window, locates the search field, types the song name, and clicks play. It is more steps, but it works for every app with zero per-app integration work.
Combining Both Approaches
The ideal setup uses both. For apps you automate frequently, dedicated MCP servers with clean APIs give the best experience. For everything else, the accessibility-based MCP server provides universal fallback coverage.
This is how Fazm approaches desktop automation. The accessibility layer gives the agent the ability to operate any app on your Mac. Specialized integrations for common workflows provide faster, more reliable automation for the tasks you run most often. The accessibility layer ensures you are never blocked because an integration does not exist yet.
Fazm is an open source macOS AI agent. Open source on GitHub.