The agent scaffolding bottleneck is a lossy pipeline, not a missing feature

The framing

When an agent does something stupid, the standard read is “the model hallucinated” or “the prompt was wrong.” Both can be true. Both miss the more common case: the model never had the information needed to do the right thing because the harness threw it away on the way in.

The harness has to throw things away. Context windows are finite, multimodal tokens are expensive, the API has hard limits on image dimensions, long context degrades instruction following. Every shipping harness against a frontier model ends up with the same shape: a stack of filters that drop, downsample, summarize, or skip content before it reaches the model.

fazm is a macOS computer-use agent on top of the Claude Agent SDK. Its harness is one Node bridge plus one Swift host process. It is open source, so the filters are countable. We counted seven filters worth talking about plus one auto-approve gate. Each is below, in source order.

Filter 1, screenshots are downsampled to 1920 px

acp-bridge/src/index.ts, lines 838 to 886

Playwright on a Retina Mac produces PNGs around 2880 by 1800. Claude’s multimodal API rejects anything past 2000 pixels on either axis. The bridge spawns a watcher on /tmp/playwright-mcp/ at startup, and any new PNG or JPEG that exceeds the cap is resampled in place by the macOS sips binary before the model has a chance to read it.

// acp-bridge/src/index.ts
const MAX_SCREENSHOT_DIM = 1920; // stay under 2000px API limit

if (w > MAX_SCREENSHOT_DIM || h > MAX_SCREENSHOT_DIM) {
  execSync(`sips --resampleHeightWidthMax ${MAX_SCREENSHOT_DIM} "${filepath}" 2>/dev/null`);
  logErr(`Screenshot resized: ${filename} from ${w}x${h} to fit ${MAX_SCREENSHOT_DIM}px`);
}

What the model loses: the actual pixel detail. A 14-pixel-tall icon at native resolution becomes a 9-pixel-tall icon after the resize, which is past the threshold where reliable text recognition stops working in current vision models. We have shipped bug reports against this filter. The trade is unavoidable, the API will reject the image otherwise. The point is not that the filter is wrong, the point is that nobody outside the bridge knows it ran.

Filter 2, Playwright never inlines a screenshot

acp-bridge/src/index.ts, line 1172

The Playwright MCP server can return screenshots one of two ways: inline, as base64 inside the tool result content array, or written to disk with only the file path returned. The bridge forces the second mode by hard-coding --image-responses omit in the launch args.

// acp-bridge/src/index.ts:1172
playwrightArgs.push(
  "--output-mode", "file",
  "--image-responses", "omit",
  "--output-dir", "/tmp/playwright-mcp",
);

What the model loses: the screenshot, as an immediate observation, on the turn that took it. To actually look at the picture the model has to spend an extra tool call (Read on the file path) on the next turn. The cost saved is real, a single 1920 px PNG is roughly 600 KB of base64, and a session that takes ten screenshots and keeps them inline blows past the cheap input window. The cost paid is one round-trip of latency on every visual decision.

Filter 3, MCP tool results are text-only

acp-bridge/src/index.ts, lines 2678 to 2714

When the bridge unwraps a tool result coming back from any MCP server, it walks the content array, copies out text items, and skips everything else. The comment in the source spells out the trade in one sentence.

// acp-bridge/src/index.ts:2679
// ACP wraps MCP content items as {type:"content", content:{type:"text"|"image", ...}}.
// We extract only text items and skip images to keep context small.
for (const item of contentArr) {
  if (item.type === "text" && typeof item.text === "string") {
    texts.push(item.text as string);
  }
  // Skip images entirely
}

What the model loses: image output from any MCP server. The macOS-use accessibility binary, for example, can return a screenshot of the window it just interacted with. The bridge drops it. So does any third-party MCP that thought it was being helpful by attaching a visual confirmation. The model sees the text summary and treats the action as opaque. On a turn where the visual would have mattered (a popup that appeared, a state change that did not), the agent continues as if nothing happened.

Filter 4, the system prompt sees the last 30 messages, no more

Desktop/Sources/Providers/ChatProvider.swift, line 1812

On every new chat session the Swift host builds a system prompt that includes a recent-conversation block. That block is hard-capped at the last 30 non-empty messages from the local store, regardless of whether the conversation is six turns or six hundred.

// Desktop/Sources/Providers/ChatProvider.swift:1812
let recent = messages
  .filter { !$0.text.isEmpty }
  .suffix(30)

// Injected into the system prompt as:
prompt += "\n\n<conversation_history>\nBelow is recent conversation history. " +
          "The user can see these messages and expects you to be aware of them. " +
          "For older conversations, query chat_messages with execute_sql.\n" +
          "\(history)\n</conversation_history>"

What the model loses: anything older than the last 30 messages, on every fresh session. The prompt does tell the model that older context exists in a SQL table it can query. The model has to know to ask. In practice, on the failure cases that show up in support, it doesn’t. It assumes the snippet it sees is the whole conversation and fills in plausible context. The user, who can scroll back forever, sees the agent as forgetful.

Filter 5, session replay caps at 20 entries, 4000 chars each

acp-bridge/src/index.ts, lines 1783 to 1810

When the agent SDK loses a session (a process restart, a crashed bridge, a recovered cold start), the bridge tries to rebuild context by replaying recent turns into a single SESSION RESTORED preamble. The replay is capped on both axes.

// acp-bridge/src/index.ts:1783
const MAX_REPLAY = 20;
const replay = ctxEntries.slice(-MAX_REPLAY);
// ...
const text = (e.text ?? "").slice(0, 4000);

What the model loses: 80 percent or more of a long conversation any time a session resets. A turn that contained the actual instruction (a 12 KB system spec the user pasted in, say) gets clipped to its first 4000 characters. The model sees a synopsis of a synopsis and continues from there. This is the layer that produces the “why did the agent restart from scratch” bug reports. The cap is what saves the recovery from being unbounded; it is also what makes the recovery dishonest.

Filter 6, skills are advertised by name, not loaded

Desktop/Sources/Providers/ChatProvider.swift, lines 1707 to 1714

The harness ships skills as separate .skill.md files in ~/.claude/skills/. The system prompt does not include the body of any of them. It includes a comma-separated list of names and a one-line instruction to fetch the body via the Skill tool when one looks relevant.

// Desktop/Sources/Providers/ChatProvider.swift:1712
prompt += "\n\n<available_skills>\nAvailable skills: \(skillNames)\n" +
          "Use the Skill tool to load full instructions for any skill before using it.\n" +
          "</available_skills>"

What the model loses: the actual instructions, until it asks. The trade is deliberate, this is what keeps the boot prompt small enough to leave room for tools and context. The cost is that “does this skill apply to what I am doing” becomes a name-only judgment, which is a worse decision than the one the model would make if it could read the body. On a skill set the model has not seen recently, the wrong one gets loaded, the work gets done in a worse way, and the user sees a slower or weirder result without knowing why.

Filter 7, attachments are gated at 20 MB and 10 MB

acp-bridge/src/index.ts, lines 1892 to 1918

User-attached files go through a MIME-and-size gate before they ever reach the model. Images and PDFs above 20 MB are rejected. Text files above 10 MB are rejected. Binary types that are not image or PDF are not inlined at all, only their path is sent.

// acp-bridge/src/index.ts:1892
const MAX_INLINE_SIZE = 20 * 1024 * 1024; // 20 MB for images/PDFs
const MAX_TEXT_SIZE   = 10 * 1024 * 1024; // 10 MB for text files

const sizeLimit = isInlineBinary ? MAX_INLINE_SIZE : MAX_TEXT_SIZE;
if (stats.size > sizeLimit) {
  // reject and tell the user to split the file
}

What the user loses: the ability to drop a 25 MB design doc, a long log file, or a recorded MP4 into the chat in one shot. The model never sees the file at all and the user gets an error. The gate is there because the API has its own per-request body limits and timing out at the API layer is a worse failure mode than a clear local rejection. It is still a filter, and the agent is dumber for it on the cases where the missing context was the whole point.

Eighth thing, not a filter, the permission gate is auto-approved

acp-bridge/src/index.ts, lines 702 to 714

The Claude Agent SDK ships with a built-in safety mechanism: when a tool call would do something risky, the SDK emits a session/request_permission ACP message and waits for the host to approve or deny. fazm answers every one of those messages with allow_always, before the user sees them.

// acp-bridge/src/index.ts:702
if (method === "session/request_permission") {
  // Auto-approve all tool permissions
  const allowAlways = options.find((o) => o.kind === "allow_always");
  const allowOnce   = options.find((o) => o.kind === "allow_once");
  const optionId = allowAlways?.optionId ?? allowOnce?.optionId ?? "allow";
  logErr(`Auto-approving permission for tool (id=${id})`);
  acpStdinWriter?.(JSON.stringify({
    jsonrpc: "2.0",
    id,
    result: { outcome: { outcome: "selected", optionId } },
  }));
}

This is not a lossy filter on the input. It is an auto-yes on the output side. Listed here because it is the same shape of decision: a piece of the SDK that is designed to slow the agent down, a deliberate choice to short-circuit it. fazm runs as a foreground app the user is watching, with a stop button visible at all times, and the design choice was that an extra modal per tool call would be unusable. Other harnesses make the opposite call. The point is that the decision is in the harness, not the model.

The leak inventory, at a glance

One card per stage. Each is small in isolation. Stack them and the model is reasoning over a heavily abridged copy of reality.

Why this is the bottleneck and not the model

Take a single Fazm turn where the user asks the agent to click a button on a web page. The Playwright MCP screenshots the page (filter 1, downsampled). Returns a path, not the bytes (filter 2, no inline). The bridge unwraps the result, drops any image content (filter 3, text-only). The model decides the screenshot is worth looking at, calls Read, gets a 1920 px PNG instead of the original 2880 px capture. It picks coordinates against the downsampled image. The action runs. If anything has gone wrong, the diagnostic dump available at the next interrupt has been truncated by filter 5 and capped by filter 4. None of this is the model failing. It is the model succeeding under conditions the harness chose for it.

Now substitute “a 10 percent smarter model” into that flow. The screenshot is still 1920 px. The image content is still dropped. The conversation history still ends at message 30. The only step the smarter model improves is the “pick coordinates” one, and even there the improvement is bounded by the resolution it was given. The next major model gives you maybe 5 percent of the win you would get by removing one filter from the pipeline.

That is the actual bottleneck. The harness is not slow, it is not buggy, it is not poorly written. It is doing exactly what it has to do given its budget. The ceiling on agent quality is the cumulative information loss across every stage of the harness, and you cannot lift the ceiling without rebuilding the stages.

What you actually do about it

Three moves, in order of how much they cost.

1. Audit your own harness, file by file. Find the constants. There will be a screenshot cap. There will be a history cap. There will be an image strip somewhere. Write them down. The list is the audit.
2. Pick the one that matches your worst failure case. If the agent forgets the conversation, the history cap is the suspect. If the agent “cannot find the button,” the screenshot cap or the image strip is the suspect. If a long task restarts from scratch, the session replay cap is the suspect. Match symptom to filter.
3. Loosen exactly that one filter and remeasure. Do not loosen the whole stack. Each filter exists for a reason and the next failure mode is waiting to bite. fazm exposes one knob (FAZM_TOOL_TIMEOUT_SECONDS) and the rest you patch in source. That is what open source is for.

On a closed harness this audit is impossible from the outside. You can guess which filter is hurting you on a given run, and that is most of what “prompt engineering” turns into in practice: working around a filter you cannot see. The argument for an open scaffolding layer is the same as the argument for an open browser engine. You eventually need to read the source.