Don't Trust Agent Self-Reports - Verify with Screenshots

Your AI agent says it sent the email. It says it filled the form. It says it clicked the right button. It is lying - or more accurately, it has no idea whether it actually succeeded.

This is the self-report trap. Language models are trained to produce confident, coherent responses. When an agent executes an action, it generates a success message because that is the most likely next token. Whether the action actually worked is a separate question the model never checks.

The Fix Is Visual Verification

After every action, take a screenshot. Compare it to what you expected. Did the page actually change? Did the form field actually get filled? Is the confirmation dialog actually showing?

This is not about sophisticated computer vision. A simple visual diff - comparing the screenshot before and after an action - catches the majority of failures. The button was clicked but nothing happened because a modal was blocking it. The text was typed but into the wrong field because focus shifted. The page navigated but hit a 404.

What We Learned in Practice

We started logging screenshot diffs after every agent action and the results were uncomfortable. About 15% of actions that agents reported as successful had no visible effect on the screen. The agent would type text into a search bar, report success, but the screenshot showed the cursor was in a different input field entirely.

The pattern is consistent - agents are overconfident about UI interactions because they operate on element references that may be stale by the time the action executes. A screenshot is ground truth. The element reference is a best guess.

Build verification into the loop, not as an afterthought. Every action should be followed by visual confirmation before the agent moves to the next step.

Fazm is an open source macOS AI agent. Open source on GitHub.