The Ghost of a Second Choice in Agent Decision Trees
The Ghost of a Second Choice
Every time an AI agent makes a decision, there is a path not taken. The agent chose to click button A instead of button B, to use tool X instead of tool Y, to try approach 1 instead of approach 2. These unchosen alternatives - the ghosts of second choices - matter more than most people realize when you are debugging a failed automation.
Why the Ghost Matters
When the chosen path fails, the agent needs to recover. The quality of that recovery depends entirely on whether it can reason about the alternatives it discarded. An agent that forgot why it chose path A cannot make a good decision about whether path B would work better.
This is the ghost: the context about why certain options were rejected. That context persists and influences future decisions even though those options were never executed - or it should, if the agent was designed to retain it.
Consider a desktop automation agent choosing how to submit a form. It sees three options:
- Click the Submit button
- Press Enter in the focused field
- Use keyboard shortcut Command+Return
The agent picks option 1. The button click fails - the element is not in a clickable state. Now the question is: why was option 2 rejected? If the agent logged that it rejected Enter because "the focused field might trigger a different action," that context tells you something important. But if the agent just recorded "clicked Submit," you are debugging blind.
The Decision Tree Problem
Agent decision trees are not like game trees where you can evaluate all branches in isolation. Each branch has side effects. The agent changed the state of the system when it took path A. Path B, which might have been viable before, may no longer work because the system state has changed.
This is especially acute in desktop automation. The agent clicked a button, which opened a dialog. Now the alternative approaches - which assumed no dialog was open - may not apply. The agent needs to know both: what alternatives were available before the action, and what the current state is after it.
Language Agent Tree Search (LATS), a framework combining Monte Carlo Tree Search with language models, addresses this directly. LATS maintains the full tree of explored and unexplored branches, uses a selection strategy that balances exploitation (going deeper on promising paths) and exploration (trying new branches when existing ones seem stuck), and crucially, preserves the rationale for branch selections throughout execution.
The practical insight from LATS is that value estimation at each branch changes as the search progresses. A branch that looked unpromising early on may become the right choice after other options have failed. Agents that discard branch information cannot do this re-evaluation.
What Good Decision Logging Looks Like
For desktop agents, this means logging not just what the agent did, but what it considered and rejected - and why.
A minimal decision log entry looks like this:
{
"step": 14,
"goal": "submit registration form",
"options_considered": [
{
"action": "click AXButton 'Submit'",
"rationale": "primary action, visible and enabled",
"chosen": true
},
{
"action": "press Enter in focused field",
"rationale": "rejected: field is a multi-line textarea, Enter adds newline",
"chosen": false
},
{
"action": "keyboard shortcut Cmd+Return",
"rationale": "rejected: not confirmed to work in this app",
"chosen": false
}
],
"outcome": "failed - element not interactable",
"state_before": "form filled, submit button visible",
"state_after": "form filled, submit button still visible, no change"
}
When the Submit click fails, you immediately know: try Cmd+Return next. The rejected option with "not confirmed to work" is the obvious recovery candidate. Without this log, you are guessing.
Compare this to the typical agent log:
Step 14: click "Submit" -> FAILED
Step 14 retry: click "Submit" -> FAILED
Step 14 retry 2: click "Submit" -> FAILED
Error: max retries exceeded
Three retries of the same failed action is not recovery. It is thrashing.
Implementing Shadow Awareness
"Shadow awareness" is the property of maintaining awareness of rejected alternatives throughout execution. There are two practical approaches:
Approach 1 - Decision context in state. The agent carries a rolling window of decision context - the last N decisions, including the options that were not taken. When selecting a recovery strategy, it queries this context.
class AgentState:
def __init__(self):
self.action_history = []
self.decision_history = [] # Includes rejected options
def record_decision(self, chosen: Action, rejected: list[RejectedOption]):
self.decision_history.append({
"chosen": chosen,
"rejected": rejected,
"timestamp": time.time(),
"system_state": self.capture_state_snapshot()
})
def get_recovery_candidates(self, failed_action: Action) -> list[Action]:
"""Find rejected alternatives from the same decision point."""
recent = self.decision_history[-10:] # Last 10 decisions
for entry in reversed(recent):
if entry["chosen"].goal == failed_action.goal:
return [r.action for r in entry["rejected"]
if r.reason != "definitely_not_applicable"]
return []
Approach 2 - Structured planning with explicit alternatives. Before executing any multi-step task, the agent produces a plan that includes alternatives for each step:
Step 3: Submit the form
Primary: click the Submit button
Fallback A: press Cmd+Return
Fallback B: navigate to /confirm endpoint directly
Give up condition: no submission mechanism found after 3 attempts
The fallbacks are planned ahead of time, before any state changes, when all options are still viable. This is cheaper than re-planning after a failure.
The Debugging Payoff
When debugging a failed automation, knowing the decision rationale is often more useful than knowing what the agent actually did. "The agent chose approach X because Y" tells you whether approach X was the right choice, whether Y was a valid reason, and whether one of the rejected alternatives would have worked.
A decision log that captures this turns debugging from archaeology into analysis. You can answer "why did this fail?" in minutes instead of hours of re-running the automation and instrumenting it with print statements.
Build agents that remember their decision rationale. The ghost of the second choice is not a problem to eliminate - it is information to preserve.
Fazm is an open source macOS AI agent. Open source on GitHub.