Hybrid AI Agent Architectures - Local Models for Sensitive Data
Hybrid AI Agent Architectures - Local Models for Sensitive Data
Your AI agent knows more about your business than most of your coworkers. It has seen your codebase, your customer data, your financial reports, and your internal communications. The question is not whether AI is useful - it is whether all of that data should leave your machine.
The Hybrid Pattern
The answer for most teams is a hybrid architecture. Route sensitive tasks to a local model running on your hardware. Route everything else to cloud models that are faster and more capable. The routing decision is based on data sensitivity, not task complexity.
Local model handles:
- Processing customer PII
- Analyzing financial documents
- Reviewing HR-related communications
- Working with proprietary algorithms or trade secrets
Cloud model handles:
- Writing documentation
- Generating boilerplate code
- Answering general technical questions
- Formatting and style tasks
Running Local Models Practically
Ollama makes local model deployment trivial on macOS. A model like Llama 3 70B runs on an M-series Mac with 64GB RAM and handles most text processing tasks competently. It is slower than Claude or GPT-4, but the data never leaves your machine.
The key insight is that sensitive data tasks rarely need the most powerful model. Extracting fields from a medical form, redacting PII from a document, or classifying internal emails by sensitivity level - these are tasks that a 7B parameter model handles fine.
The Routing Layer
Build a simple router that classifies each task before sending it to a model. Check for PII patterns, file paths to sensitive directories, or explicit sensitivity tags. When in doubt, route locally. The performance hit of using a local model for a non-sensitive task is minor. The cost of sending sensitive data to the cloud is potentially catastrophic.
The Uncomfortable Truth
Most teams send everything to cloud APIs because it is easier. The hybrid approach requires more infrastructure and more thought about data classification. But when a data breach happens, "it was easier to send everything to the cloud" is not a defense anyone wants to make.
Fazm is an open source macOS AI agent. Open source on GitHub.