A Generally Adopted Benchmark for Local AI Inference Speed
A Generally Adopted Benchmark for Local AI Inference Speed
"It runs fast on my machine" is not a benchmark. When evaluating hardware for local AI agent workloads, you need reproducible numbers. llama-bench provides exactly that - a standardized way to measure tokens per second across different models and hardware configurations.
Why tok/s Matters for Agents
For a desktop AI agent, inference speed directly affects user experience. An agent that generates 30 tokens per second feels responsive. One that generates 5 tokens per second feels sluggish. The difference between a useful agent and an annoying one is often just hardware and model selection.
Tokens per second also determines how many agent actions you can chain before the user gets impatient. A fast inference setup lets you run multi-step workflows - plan, execute, verify, correct - in the time a slow setup takes for a single step.
Running llama-bench
llama-bench runs standardized prompt and generation benchmarks against any GGUF model file. It reports prompt processing speed (tokens per second for input) and text generation speed (tokens per second for output) separately, which matters because they have different performance characteristics.
The numbers are reproducible. Run the same model on the same hardware twice and you get the same results. This means you can compare your M2 MacBook results against someone else's M3 Max results and the comparison is meaningful.
What the Numbers Tell You
For desktop agent workloads, generation speed matters most. An agent spends most of its time generating action plans and responses, not processing long prompts. Look for at least 20 tok/s on your target model size for a responsive experience.
Prompt processing speed matters for initial context loading - feeding the agent its system prompt and conversation history. Slower prompt processing means longer startup time but does not affect interactive response speed.
Having a standard benchmark eliminates the guesswork. Before buying hardware for AI agent workloads, run llama-bench with the models you plan to use. The numbers do not lie.
Fazm is an open source macOS AI agent. Open source on GitHub.