Why AI Desktop Agents Need Granular Security Policies

A post in r/ClaudeCode introduced HushSpec - an open specification for security policy at the action boundary of AI agents. It hit a nerve because the current state of agent permissions is basically "allow everything" or "allow nothing," and neither works in practice.

When you grant an AI agent access to the macOS Accessibility API, you are giving it the ability to click buttons, read screen content, and type text in any application. That is an enormous amount of power. The question is not whether to give agents this power - it is how to scope it properly.

Per-App, Per-Action Rules

The HushSpec approach defines security policies at the granular level. Instead of a single toggle for "can this agent use Accessibility API," you define rules like:

• Agent can read text from any app but only click buttons in Slack and Calendar
• Agent can type in text fields in Chrome but cannot interact with the address bar
• Agent can open files in Finder but cannot delete them

This mirrors how mobile operating systems handle permissions - not "allow all sensors" but "allow camera in this app, allow location only while using." Desktop agents need the same granularity.

Why This Matters for Trust

The biggest blocker to AI agent adoption is trust. People will not let an agent manage their desktop if one miscalibrated action could send a wrong email or delete a file. Bounded tools and approval flows help, but they need to be backed by a policy layer that enforces limits even when the agent's reasoning goes wrong.

A well-defined security policy also makes agents auditable. When every action passes through a policy check, you get a complete log of what the agent attempted, what it was allowed to do, and what was blocked. This is essential for teams deploying agents in professional environments where secret management and data access controls matter.

The bottom line: binary permissions are a dead end. AI agents need policy engines that understand the difference between reading a spreadsheet and editing one.

Fazm is an open source macOS AI agent. Open source on GitHub.