A local AI desktop agent that watches first, proposes one task, then drives any app on your Mac

Two things. First, the control plane runs on your Mac: a Swift app holds the permission state, a Node-based ACP bridge invokes tools, a Claude Code subprocess does the thinking, and an MCP server calls macOS accessibility APIs on your real apps. Nothing routes through a server to get back to you. Second, the trigger is local: a rolling 60-minute active-window video buffer is analyzed by Gemini in the background to propose one concrete task you could hand to the agent. The inference endpoints themselves (Claude, Gemini) are still cloud-hosted, but every byte of state about what you are doing lives on your machine until you accept a suggestion.

GeminiAnalysisService.swift:69 in the public Fazm source sets targetDurationSeconds: TimeInterval = 3600 with the inline comment 'Trigger analysis after 60 min of recordings'. The same file, line 68, caps the rolling buffer at maxChunks = 120 (which translates to 120 roughly 30-second focused-window clips). The model used for analysis is gemini-pro-latest (line 67). These are literal Swift constants, not marketing language.

The prompt at GeminiAnalysisService.swift:12 asks Gemini for the ONE most impactful task an AI agent could take off the user's plate, and forces it to run two queries before deciding: a SELECT against observer_activity to avoid proposing work it already proposed, and a SELECT against chat_messages to avoid proposing work the user is already having the agent do. If the suggestion overlaps with a previous task's app and category, the verdict returned is NO_TASK. If the video is too blurry or the user's intent is unclear, the verdict is UNCLEAR. Only a high-confidence match returns TASK_FOUND with a description and a markdown recommended-approach document.

Fazm spawns a Claude Code subprocess. That subprocess has shell access, file system access, and a registered set of MCP servers (macos-use for accessibility-driven app control, whatsapp, google-workspace, playwright, and any you install). It also has access to the skills in ~/.claude/skills/ that the Fazm installer dropped there on first launch. When the agent decides it needs to click a button in Mail or type into Notes or drive a browser, it does that through the accessibility tree exposed by the macos-use MCP server. The result is a desktop agent that can touch any Mac app that exposes AX, not just the browser.

SkillInstaller.swift auto-discovers every *.skill.md file in Desktop/Sources/BundledSkills/ and copies it into ~/.claude/skills/<name>/SKILL.md. The current list is: ai-browser-profile, canvas-design, deep-research, doc-coauthoring, docx, find-skills, frontend-design, google-workspace-setup, pdf, pptx, social-autoposter, social-autoposter-setup, telegram, travel-planner, video-edit, web-scraping, xlsx. Each is a markdown file with YAML frontmatter that Claude Code reads at the start of a session. That means your 'agent' does not rely on a black-box policy. You can open the file, read what it does, edit it, delete it. The installer runs a SHA-256 checksum compare on every launch and only overwrites files whose bundled version actually changed.

No. The Claude Code subprocess uses your Anthropic account or subscription to hit api.anthropic.com. The observer's task-discovery analysis calls Gemini via Google's Generative AI endpoint. The 'local' part is that the agent loop, the permission state, the screen recording buffer, the active-window metadata, the fazm.db SQLite database, and the entire set of tools run on your Mac. No proxy server in between. If you prefer on-device inference, the Claude Code side can be pointed at a compatible local endpoint; the observer currently depends on Gemini's multimodal video input.

Those three are prompt-driven: you type an instruction, the agent screenshots, the model reads pixels, it clicks, repeat. Fazm flips that direction. It watches silently and proposes work, which means you get suggestions for tasks you would not have thought to ask about. It also uses the macOS accessibility tree as the primary input for actions, not screenshots, which is faster and gives the model a structured role+label tree rather than raw pixels. Finally, Fazm is shipped as a consumer Mac app with a one-click installer, not a framework or a playground. There is no SDK step.

Active-window video chunks are written to Application Support on your Mac (chunksDir in GeminiAnalysisService.swift, line 81) and survive app restarts via a JSON buffer index. Older screenshots are cleaned up after a configurable retention window in ScreenCaptureManager.swift. The discovered tasks themselves are stored in the observer_activity table inside fazm.db, which is a local SQLite file. When analysis fires, the relevant chunk files are uploaded to Gemini for multimodal inference and then remain on disk until the retention cleanup runs or the buffer wraps.

Yes, but the proactive-task observer is the thing that needs screen recording plus accessibility. Without screen recording, Fazm still works as a reactive agent: you can open the floating bar with Cmd+\, hit your configured Ask Fazm shortcut, and have the agent drive apps for you. Without accessibility, the agent cannot click buttons in external apps and falls back to shell and filesystem operations only. The two permissions unlock different modes, and the onboarding flow walks through each separately.

The analysis prompt at GeminiAnalysisService.swift:30-34 makes two mandatory SELECT queries before deciding: one against observer_activity (the discovered_tasks log) and one against chat_messages (the user's recent agent conversations). The instruction is explicit: 'If the observer_activity query returns tasks that are similar to what you would suggest (same app, same type of work, same general activity), return NO_TASK.' It goes further and treats same-app same-category as similar, even when the details differ, and errs toward NO_TASK when in doubt. The result is that after the first week, you get fewer suggestions, not more, because the system compounds its knowledge of what you have already been offered.

Apps that expose the macOS accessibility tree work out of the box through the generic macos-use MCP server (click_and_traverse, type_and_traverse, scroll_and_traverse, press_key_and_traverse, refresh_traversal). That covers nearly every native SwiftUI, AppKit, and Catalyst app. The soft spots are Electron apps with partial AX trees, some Chromium-based tools, and very old pre-notarized apps. For those, Fazm ships specific MCP servers that bypass AX: playwright for browser automation, whatsapp for the WhatsApp Catalyst app, google-workspace for REST-based Google surfaces. If an app genuinely exposes no AX and no documented API, the agent falls back to shell commands or explains the limit instead of hallucinating an action.

Yes. The Fazm desktop source tree is public. The 60-minute constant is at Desktop/Sources/GeminiAnalysisService.swift line 69. The 120-chunk cap is at line 68. The bundled skill installer is at Desktop/Sources/SkillInstaller.swift. The 17 bundled skills are files in Desktop/Sources/BundledSkills/. The accessibility probe code (Finder fallback + CGEvent tap) is at Desktop/Sources/AppState.swift lines 468 and 490. You can build the app yourself and verify that the observer actually runs the rolling buffer the way this page describes. The consumer app is free to install; the paid tier exists for extended usage and priority support.