How to Build AI Agents You Can Actually Trust

The single biggest lesson from building a desktop AI agent: giving the agent broad system access is a recipe for disaster.

Early versions of Fazm had relatively open access to macOS APIs. The agent could click anything, type anywhere, open any app. It worked great in demos. In real use, it occasionally did things like closing the wrong window or sending a half-drafted email. The why local-first matters argument becomes even stronger when you consider trust - keeping data processing local reduces the blast radius of agent mistakes.

Bounded Specialized Tools

The fix was moving every action through a typed tool interface. Instead of "click at coordinates (x, y)", the agent calls press_button(app: "Safari", element: "Submit"). Instead of "type this text", it calls fill_field(app: "HubSpot", field: "Deal Notes", value: "...").

This gives you:

• Type safety. The agent cannot click a button that does not exist or type into a read-only field.
• Audit trail. Every action is a structured call with named parameters. You can log and review everything the agent did.
• Permission boundaries. You can allow the agent to read from any app but only write to specific ones.

The Approval Fatigue Problem

The obvious safety mechanism is approval: the agent proposes an action, you approve it. But if the agent asks for approval on every single click, the UX is worse than just doing the task yourself. This is approval fatigue, and it kills agent adoption.

The solution is batching related actions into plans that get approved once:

Plan: Update CRM with call notes
1. Open HubSpot
2. Navigate to Acme Corp deal
3. Add note: "Call with Sarah - discussed Q2 timeline..."
4. Update stage to "Negotiation"
5. Set follow-up date to March 20

[Approve] [Edit] [Cancel]

One approval for five actions. The user sees exactly what will happen, can edit any step, and approves the entire sequence at once. This is the sweet spot between safety and usability. These are exactly the kind of cross-app workflows where batched approval shines - multiple apps, one confirmation.

Accessibility Tree as a Safety Layer

Using the accessibility tree for UI interaction is not just a reliability choice - it is also a safety choice. The accessibility tree tells you exactly what each element does. A button labeled "Delete All" is clearly destructive. A button labeled "Save" is clearly safe.

When the agent works through the accessibility tree, you can build policies like "never click elements with destructive labels without explicit approval" directly into the tool interface. With screenshot-based agents, you would need a vision model to make that judgment call, which adds latency and uncertainty. Our comparison of DOM control vs screenshots explains why structured access is more reliable for this kind of policy enforcement.

---

Fazm uses bounded tools and plan-based approval for safe desktop automation. Open source on GitHub. Inspired by discussions in r/AI_Agents.