Local AI privacy beyond inference

The frame people get wrong

The shorthand "local AI" came out of the local-LLM community, where the question was always: did you run the weights on your hardware, or did you call somebody else's API. That frame is correct for an isolated inference workload. It is incomplete for a desktop agent because an agent is not just a model. It is a model plus a microphone, plus a screen reader, plus a clicker, plus a telemetry pipe, plus an auth layer, plus a crash reporter. Each of those is its own data path. Each one has a default destination, and the defaults are almost never "your Mac."

A useful exercise is to grep an open-source agent for every hard-coded URL the binary can reach. For Fazm the count is small enough to enumerate by hand. There are five hardcoded external hostnames in the Swift source, plus a sixth (the LLM endpoint) that is read from an environment variable. That is the entire externally-talking surface of the app. The list below is what each of those endpoints does, why it is there, and what you can do about it.

Verified against the open MIT source at github.com/m13v/fazm, re-checked 2026-05-10. Hostnames live in TranscriptionService.swift, ChatToolExecutor.swift, PostHogManager.swift, AuthService.swift, and ACPBridge.swift.

Surface 1, model inference

The one most readers think about. Default behavior in most agents is a cloud LLM call on every conversation turn. Fazm is no different out of the box: the chat subprocess talks to api.anthropic.com unless you override it. The override is a single environment variable, ANTHROPIC_BASE_URL, which the agent subprocess reads at launch. The assignment lives in Desktop/Sources/Chat/ACPBridge.swift around line 468. The Settings UI surfaces it as the "Custom API Endpoint" field.

Three useful values to put in that field, in order of how local you want to be. Your corporate proxy if you have one. AWS Bedrock or Vertex AI in your region if you need a specific data residency. http://127.0.0.1:11434 or wherever your Ollama, LM Studio, or llama-server lives if you want fully local inference. Once you set it, nettop should show no api.anthropic.com row while the agent is working.

Surface 2, voice transcription

The loud one most local-first writeups skip. A voice-first agent has a microphone hot during conversation, and turning audio into text is the work of a transcription service. Fazm uses Deepgram by default. The endpoints are hardcoded at TranscriptionService.swift:276 (wss://api.deepgram.com/v1/listen for streaming) and TranscriptionService.swift:561 (https://api.deepgram.com/v1/listen for batch). When you talk, the audio goes to Deepgram in real time. That is true even if your LLM endpoint is on 127.0.0.1.

Two ways to fix it. Turn voice off and use text input. Or run an on-device transcriber (whisper.cpp with metal acceleration, MLX-Whisper, or WhisperKit) and route the agent's audio there instead of Deepgram. Either way, the verification is the same: while you talk, nettop should show no Deepgram row.

Surface 3, screen capture

Screenshot agents are the most data-heavy approach to desktop control. Every step sends a full-resolution image of your display (or a focused window) to a vision model. The pixel volume is enormous compared to anything else the agent does, and the bytes contain whatever was on screen at that moment, including content from apps the user did not consciously include in the request.

Fazm has a screen-capture path (Desktop/Sources/FloatingControlBar/ScreenCaptureManager.swift, using CGWindowListCreateImage) but it is not the default control loop. It is exposed as the capture_screenshot tool that the model can call when it explicitly needs visual confirmation, e.g. to verify a UI state that is not in the accessibility tree. Most steps do not call it. The default surface for "what is on the user's screen" is the accessibility tree, which is much smaller and much harder to use to reconstruct private content.

Surface 4, accessibility tree

The structured alternative to pixels. macOS exposes every UI element as an AXUIElement with attributes (role, title, value, position, children). The agent walks that tree to find the right button and fires a click at its computed coordinates. The data the model sees is text and structure, not images. The tree is read on-device by the OS, and what you ship to the LLM is whatever subset is relevant to the current step.

The privacy implication is one most readers underweight. A pixel-based agent has to send the full visual context every time. A tree-based agent sends the minimum slice needed to act. If the model is local, that slice never leaves your machine at all. If the model is cloud-hosted, the slice is dramatically smaller and harder to reconstruct screenshots from. Same task, less data on the wire, fewer privacy questions to argue about.

Surfaces 5 through 8, the quiet ones

The remaining surfaces ship by default in almost every consumer Mac app. They are individually small but collectively they decide whether a vendor knows you exist, what version you ran, when you opened the app, and which feature you used.

• Telemetry. Fazm uses PostHog. The host is hardcoded at PostHogManager.swift:14 as us.i.posthog.com. Events fire on app open, agent run, settings change, etc. The property bag never includes user content (chat text, file paths, accessibility tree contents) by design. The destination is a single host you can grep for, watch in nettop, and block at the firewall if you want to.
• Crash reports. Sentry SDK. SentrySDK.setUser is called in AuthService.swift:735-738 with a user id and email so a crash report can be attributed. No screen content, no audio, no AX tree contents. If you opt out, no crash reports are sent.
• Auth and identity. Firebase. The endpoints are accounts.google.com/o/oauth2 (sign-in), oauth2.googleapis.com/token (token exchange), identitytoolkit.googleapis.com (account state), securetoken.googleapis.com (refresh), and firestore.googleapis.com (account-scoped state). These fire when you sign in and when your token refreshes, not continuously. If you sign in with Google or Apple, your identity goes to Firebase. The agent itself does not need an account to run.
• Browser cookies. When the agent drives Chrome via Playwright, it is using your real browser session and your real cookies. That is a feature (the agent can do work in your logged-in apps) and a privacy surface (the model gets to see whatever a logged-in tab shows). Treat the agent's browser actions as actions you took yourself, because from the third-party site's point of view that is exactly what they are.

A two-command sanity check

The shortest possible audit. Run nettop scoped to the agent process while you exercise the feature you care about. Then read its TCC row to see which permissions it actually holds. Anything you cannot explain is a surface worth investigating.

The shape of a "mostly local" agent

You do not need to make every surface fully local to dramatically reduce the data picture. Three changes get you most of the way for a modest capability hit. Set the LLM endpoint to a local server (or to a private gateway in your region). Turn voice off, or repoint it to an on-device transcriber. Opt out of analytics if the build exposes the toggle. After those three, the surfaces that survive are auth (one-time at sign-in), crash (rare, opt-in), and the actions you actually asked the agent to take. That is a defensible answer to "where does my data go," and it is much more useful than a binary local-or-not flag.

The reason this works is that the surfaces are independent. Fixing inference does not fix voice, fixing voice does not fix telemetry, and so on. The corollary is that any agent that lets you fix only one of them and calls itself "private" is using the word loosely. Walk the eight surfaces, decide each one, then ship.