On-Device AI on Apple Silicon

Apple Silicon changed what is possible for local AI. The unified memory architecture means ML models can run on the GPU without copying data between CPU and GPU memory. For a desktop agent that needs to process screen content in real-time, this matters a lot.

What Runs Locally Now

On an M1 with 16GB of RAM, you can comfortably run:

• WhisperKit for voice transcription - fast enough for real-time push-to-talk
• Ollama with 7-13B parameter models for action planning - usable latency for simple tasks
• Vision models for screen understanding - when accessibility APIs are not enough

On an M4 Pro with 48GB, the picture gets much better:

• 32B parameter models run at interactive speeds
• Multiple models simultaneously - transcription and planning can run in parallel without contention
• Overnight batch processing - the agent can process files, organize documents, and handle backlog tasks while you sleep

The Latency Question

Cloud APIs add 500ms-2s per request. For a desktop agent that might need 5-10 LLM calls to complete a single task, that is 5-20 seconds of waiting. Local inference on Apple Silicon cuts this to near-zero for smaller models.

The tradeoff is accuracy. A local 13B model is not as capable as Claude for complex multi-step reasoning. But for straightforward desktop automation - filling forms, navigating menus, extracting text - smaller models are usually sufficient. Our post on how LLMs control your computer covers the full architecture of voice-driven, local-first desktop agents.

The Privacy Argument

A desktop agent sees everything on your screen. Every password, every private message, every financial document. Running the AI model locally means none of that data leaves your machine.

This is not a theoretical concern. Screenshot-based cloud agents literally upload images of your screen to remote servers every few seconds. If your screen shows your bank account, that screenshot is now on someone else's server.

Local inference eliminates this entirely. Your screen content stays in your RAM, gets processed by your GPU, and the results stay on your machine. We make the full case for this architecture in why local-first AI agents are the future.

How Fazm Uses Apple Silicon

Fazm is designed to take advantage of Apple Silicon's unified memory:

• Voice input goes through WhisperKit locally
• Screen capture uses ScreenCaptureKit (hardware-accelerated) - see our deep dive into ScreenCaptureKit and accessibility APIs for implementation details
• You choose between local models via Ollama or cloud models like Claude
• The accessibility tree is processed entirely on-device

The goal is that the most privacy-sensitive operations - capturing your screen and understanding your voice - always happen locally, regardless of which LLM you use for action planning.

---

Fazm is open source on GitHub. Discussed in r/macapps.