The Asymmetric Trust Problem - When Your AI Agent Has More Access Than You Intended
The Asymmetric Trust Problem - When Your AI Agent Has More Access Than You Intended
When you enable accessibility permissions for an AI agent on macOS, you are granting access to an API designed for screen readers. Screen readers need to see everything - every button label, every text field value, every menu item, every piece of content on screen. That is the whole point.
The problem is that "everything" really means everything. Your password manager showing a credential in a text field. Your banking app displaying account balances. Your email client with the full body of every message. The accessibility API exposes all of it as structured, machine-readable data.
How Wide the Access Actually Is
Research into macOS permission patterns reveals a concrete scale problem: the average macOS user grants 3.7 accessibility permissions to applications - but only 1.2 of those permissions are actively used per week. Most users do not track which apps have accessibility access, and macOS does not show a running log of which app read what from the accessibility tree when.
The TCC (Transparency, Consent, and Control) framework that manages accessibility permissions was built for a world where the consumer of those permissions was an assistive technology operated by a human. The human could see what the screen reader was narrating. They knew what was being accessed because they were sitting in front of it.
An autonomous AI agent changes that model entirely. The agent can read content that is not in the active window, from applications that are not focused, at times when the user is not watching. The constraint on what it reads is behavioral - what the system prompt says - not technical.
The Trust Gap in Practice
When a user grants accessibility permissions, their mental model is usually "this app can control my mouse and keyboard." That model is incomplete in two important ways.
First, the read access is much broader than mouse and keyboard control suggests. The app can read every piece of text displayed anywhere on the system, understand the structure of every application's UI, and potentially access values in secure text fields depending on how the application implements them.
Second, the constraints on what the agent does with this access are enforced by software policy, not system architecture. A system prompt that says "only interact with Finder" is instructions, not a technical lock. The agent's compliance with those instructions depends on the quality of the model and the robustness of the agent implementation.
This asymmetry - between the access the user believes they granted and the access they actually granted - is the core problem.
Real Vulnerabilities in the Ecosystem
The TCC framework has also been demonstrated to have exploitable bypass vulnerabilities. CVE-2024-44133, called "HM Surf," involved removing TCC protection for Safari's browser directory to access camera, microphone, and location data without user consent. It was patched in macOS Sequoia 15, but it illustrates that TCC is not impenetrable.
A separate vulnerability in the com.apple.scrod MIG service - designed to handle screen reader operations - carried powerful TCC entitlements including access to Apple Events, the microphone, and sensitive documents. Attackers who could inject into trusted processes could inherit those entitlements. This class of attack specifically targets the accessibility stack.
These are not theoretical exploits. They affect real systems and were significant enough to receive CVEs and patches.
The Scoping Problem
You cannot easily scope accessibility permissions to specific applications in standard macOS. The dialog says "Do you want to allow [App] to control your computer?" It does not say "for which applications" or "reading what data." macOS grants system-wide access or nothing.
This was an acceptable design when assistive technologies were the only consumers. Assistive technologies were used by people with disabilities who needed comprehensive access to everything they were working with, and the technologies were reviewed and distributed through predictable channels.
It is a less acceptable design when the consumer is an autonomous agent that may be running background workflows, processing content from multiple applications simultaneously, and making decisions about what to read without the user watching.
Practical Mitigation Strategies
The honest answer is that this is an unsolved problem at the system level. Apple has not updated the accessibility permission model to accommodate autonomous agents. Until they do, the mitigations are behavioral:
Minimize your screen surface during agent sessions. Close applications that contain sensitive information you are not using for the current task. If the agent is helping with Xcode, your password manager and banking app do not need to be open.
Use a dedicated user account for agent workflows. A secondary macOS user account that does not have your email, password manager, or financial apps configured limits what the agent can see even if it reads beyond its intended scope. This is inconvenient but substantially reduces exposure.
Prefer agents that log their accessibility tree reads. Some agents publish a log of which applications they read from during a session. Reviewing this log after a session tells you whether the agent stayed within expected scope. An agent that cannot produce this log provides no way to verify its behavior.
Audit accessibility permissions periodically. Check System Settings > Privacy & Security > Accessibility and remove any applications that no longer need access. Permission accumulation is real - most users grant access when asked and never revoke it.
Understand the system prompt is not a technical constraint. If you are evaluating an agent for sensitive workflows, understand that behavioral constraints are only as strong as the model's compliance. Test this explicitly by asking the agent to read from applications outside its declared scope.
What Would Actually Help
A proper solution would involve per-application scoping for accessibility permissions - granting an agent access to Xcode without granting it access to 1Password, even if both are running. This is technically feasible; macOS already distinguishes between applications in other permission categories. It has simply not been built for accessibility.
A companion improvement would be read logging that users can inspect - similar to how iOS shows camera and microphone access indicators. Knowing that an agent read from your email client during a session would at least make the access visible, even if it does not prevent it.
Until these system-level improvements exist, the responsibility falls on users to structure their environments conservatively and on agent developers to build transparent, auditable systems.
Fazm is an open source macOS AI agent. Open source on GitHub.