The accessibility Crate: Using AXUIElement from Rust on macOS

macOS exposes every application's UI through the Accessibility framework. The core type is AXUIElement, a Core Foundation object that represents a single node in an app's accessibility tree. Apple provides C and Objective-C bindings. If you want to use it from Rust, the accessibility crate gives you safe, idiomatic wrappers around those raw Core Foundation calls.

This guide covers everything you need to go from cargo add accessibility to reading UI trees, querying element attributes, and performing actions like pressing buttons or typing into text fields.

Why Rust for macOS Accessibility

Swift and Python are the obvious choices for scripting AXUIElement. They work fine for small tools. But when you are building something that reads the accessibility tree thousands of times per second (a desktop automation agent, a screen reader, a testing harness), Rust gives you three things the scripting languages do not:

1. Zero-cost FFI to Core Foundation. The accessibility crate wraps AXUIElementCopyAttributeValue, AXUIElementPerformAction, and friends directly. No runtime bridging, no garbage collector pauses.
2. Deterministic memory management. AXUIElement objects are Core Foundation types with manual retain/release semantics. Rust's ownership model maps cleanly onto this, so you do not leak refs or double-free them.
3. Thread safety guarantees at compile time. The Accessibility API is not thread-safe. Rust's Send and Sync traits catch mistakes before they become 2 AM segfaults.

| Language | FFI overhead | Memory model | Thread safety | Best for | |---|---|---|---|---| | Swift | None (native) | ARC | Runtime checks | GUI apps, small scripts | | Python (pyobjc) | Bridge per call | GC | GIL protects | Quick prototyping | | Rust (accessibility crate) | None (raw CFfi) | Ownership | Compile-time | High-frequency automation, agents | | C/ObjC | None | Manual | Manual | System frameworks |

Setting Up the Crate

Add the dependency to your Cargo.toml:

[dependencies]
accessibility = "0.1"
accessibility-sys = "0.1"  # only if you need raw bindings

The crate re-exports types from accessibility-sys, which provides the raw AXUIElement FFI bindings generated from the macOS Accessibility headers. Most of the time you will only interact with the higher-level accessibility crate.

Core Concepts: The AXUIElement Tree

Every running application on macOS exposes an accessibility tree. The root is the application element. Below it you find windows, then toolbars, buttons, text fields, tables, and so on. Each element has attributes (role, title, value, position, size) and may support actions (press, increment, show menu).

The accessibility crate models this as a struct wrapping the raw AXUIElementRef. You get the system-wide element or a per-app element, then walk the tree by querying children.

Getting the System-Wide Element

The system-wide element is your entry point when you do not know which app you care about. It represents the entire screen:

use accessibility::AXUIElement;

fn main() {
    // The system-wide element lets you query the focused element
    // across all applications
    let system = AXUIElement::system_wide();

    // Get whatever element currently has keyboard focus
    if let Ok(focused) = system.focused_uielement() {
        println!("Focused element role: {:?}", focused.role());
        println!("Focused element title: {:?}", focused.title());
    }
}

When you know the target app, create an application element from its PID:

use accessibility::AXUIElement;

fn element_for_app(pid: i32) -> AXUIElement {
    AXUIElement::application(pid)
}

You can get the PID from NSRunningApplication or by shelling out to pgrep. The accessibility crate does not handle process discovery; that is a separate concern.

Reading Attributes

Every AXUIElement has attributes you can query. The most common ones:

| Attribute | Type | What it tells you | |---|---|---| | kAXRoleAttribute | String | Element type: AXButton, AXTextField, AXWindow | | kAXTitleAttribute | String | Display label ("Save", "File", "Untitled") | | kAXValueAttribute | Any | Current value (text content, checkbox state, slider position) | | kAXPositionAttribute | AXValue (CGPoint) | Screen coordinates of top-left corner | | kAXSizeAttribute | AXValue (CGSize) | Width and height in points | | kAXChildrenAttribute | Array | Child elements in the tree | | kAXEnabledAttribute | Bool | Whether the element accepts interaction | | kAXFocusedAttribute | Bool | Whether the element has keyboard focus |

The crate provides typed accessors for common attributes:

use accessibility::AXUIElement;

fn inspect_element(element: &AXUIElement) {
    // Typed accessors return Result types
    match element.role() {
        Ok(role) => println!("Role: {}", role),
        Err(e) => println!("Could not read role: {:?}", e),
    }

    if let Ok(title) = element.title() {
        println!("Title: {}", title);
    }

    if let Ok(children) = element.children() {
        println!("Child count: {}", children.len());
        for child in &children {
            if let Ok(role) = child.role() {
                print!("  - {}", role);
            }
            if let Ok(title) = child.title() {
                print!(" \"{}\"", title);
            }
            println!();
        }
    }
}

For attributes not covered by typed accessors, use the generic attribute method with the raw attribute name string.

Walking the Full Tree

To read the entire accessibility tree of an application, walk it recursively via the children attribute:

use accessibility::AXUIElement;

fn walk_tree(element: &AXUIElement, depth: usize) {
    let indent = "  ".repeat(depth);
    let role = element.role().unwrap_or_default();
    let title = element.title().unwrap_or_default();

    if title.is_empty() {
        println!("{}{}", indent, role);
    } else {
        println!("{}{} \"{}\"", indent, role, title);
    }

    if let Ok(children) = element.children() {
        for child in &children {
            walk_tree(child, depth + 1);
        }
    }
}

fn main() {
    let pid = 12345; // replace with actual PID
    let app = AXUIElement::application(pid);
    walk_tree(&app, 0);
}

A typical Finder window produces 50 to 200 nodes. A complex app like Xcode can produce thousands. If you are polling the tree repeatedly (for an automation agent), limit the depth or filter by role to keep latency under control.

Performing Actions

Reading the tree is only half the story. AXUIElement also lets you perform actions: pressing buttons, setting text field values, opening menus, and more.

use accessibility::AXUIElement;

fn press_button(element: &AXUIElement) -> Result<(), accessibility::Error> {
    // AXPress is the standard action for buttons
    element.perform_action("AXPress")
}

fn set_text_field_value(element: &AXUIElement, text: &str) -> Result<(), accessibility::Error> {
    // First focus the field, then set its value
    element.set_attribute("AXFocused", true.into())?;
    element.set_attribute("AXValue", text.into())?;
    Ok(())
}

Common actions and when to use them:

| Action | Elements | Effect | |---|---|---| | AXPress | Buttons, checkboxes, menu items | Activates the control | | AXShowMenu | Menu bar items, popup buttons | Opens the dropdown | | AXRaise | Windows | Brings the window to front | | AXConfirm | Text fields, combo boxes | Confirms the current value | | AXCancel | Sheets, dialogs | Dismisses without saving | | AXIncrement / AXDecrement | Sliders, steppers | Adjusts the value |

Finding Elements by Role and Title

In practice, you rarely want the entire tree. You want "the button labeled Save" or "the text field with the placeholder Search". Write a helper that searches by role and title:

use accessibility::AXUIElement;

fn find_element(
    root: &AXUIElement,
    target_role: &str,
    target_title: &str,
) -> Option<AXUIElement> {
    let role = root.role().unwrap_or_default();
    let title = root.title().unwrap_or_default();

    if role == target_role && title == target_title {
        return Some(root.clone());
    }

    if let Ok(children) = root.children() {
        for child in &children {
            if let Some(found) = find_element(&child, target_role, target_title) {
                return Some(found);
            }
        }
    }

    None
}

fn main() {
    let app = AXUIElement::application(12345);
    if let Some(button) = find_element(&app, "AXButton", "Save") {
        let _ = button.perform_action("AXPress");
        println!("Pressed Save");
    }
}

For production use, consider caching element references. But be aware that AXUIElement refs can go stale when the app redraws its UI. Always handle errors from attribute queries gracefully.

The accessibility-sys Crate

The accessibility crate builds on accessibility-sys, which provides raw FFI bindings to the macOS Accessibility C API. If you need something the high-level crate does not expose (parameterized attributes, observer callbacks, custom action definitions), drop down to accessibility-sys:

use accessibility_sys::{
    AXUIElementCopyAttributeValue,
    AXUIElementPerformAction,
    kAXErrorSuccess,
};
use core_foundation::string::CFString;
use core_foundation::base::TCFType;

// Raw attribute query when the high-level API doesn't cover your case
unsafe fn raw_attribute_query(
    element: accessibility_sys::AXUIElementRef,
    attr: &str,
) -> Option<core_foundation::base::CFTypeRef> {
    let attr_name = CFString::new(attr);
    let mut value: core_foundation::base::CFTypeRef = std::ptr::null();
    let err = AXUIElementCopyAttributeValue(
        element,
        attr_name.as_concrete_TypeRef(),
        &mut value,
    );
    if err == kAXErrorSuccess as i32 && !value.is_null() {
        Some(value)
    } else {
        None
    }
}

You should almost never need this. The high-level crate covers the common cases.

Common Pitfalls

• Forgetting Accessibility permissions. The most common "it does not work" issue. Your process needs to be in the Accessibility allow list. There is no runtime prompt on macOS for accessibility (unlike camera or microphone). The user must manually add your app in System Settings. Detect this early with AXIsProcessTrusted() and show a clear error message.

• Stale element references. An AXUIElement reference points to a specific UI element. If the app rebuilds its view hierarchy (navigating to a new screen, closing a sheet, reloading data), your reference becomes invalid. Always re-query from the window or application level rather than caching deep tree references across interactions.

• Blocking the main thread with tree walks. Walking a large accessibility tree takes 10 to 50ms for a typical app, but can spike to 200ms or more for complex apps like Xcode or Chrome. If your tool has a UI, do tree walks on a background thread. Just remember that AXUIElement is not Send, so you need to create the element on the thread that will use it.

• Ignoring error codes. The Accessibility API returns specific error codes: kAXErrorCannotComplete (app is busy), kAXErrorNotImplemented (element does not support this attribute), kAXErrorInvalidUIElement (stale reference). Handle each one differently rather than treating all errors the same.

• Assuming all apps implement accessibility correctly. Electron apps, games, and some older Cocoa apps have broken or minimal accessibility trees. Your tree walker might find an AXWindow with zero children, or buttons with no title. Build your tool to handle sparse trees gracefully.

Minimal Working Example

A complete program that finds the frontmost app, reads its first window's children, and prints them:

use accessibility::AXUIElement;

fn main() {
    // Check accessibility permissions
    let trusted = accessibility::AXIsProcessTrusted();
    if !trusted {
        eprintln!("This process needs Accessibility permissions.");
        eprintln!("Add your terminal to System Settings > Privacy > Accessibility");
        std::process::exit(1);
    }

    // Get the focused application
    let system = AXUIElement::system_wide();
    let focused_app = match system.focused_application() {
        Ok(app) => app,
        Err(e) => {
            eprintln!("Could not get focused app: {:?}", e);
            return;
        }
    };

    let app_title = focused_app.title().unwrap_or_else(|_| "Unknown".into());
    println!("Focused app: {}", app_title);

    // Get windows
    if let Ok(windows) = focused_app.windows() {
        for (i, window) in windows.iter().enumerate() {
            let title = window.title().unwrap_or_else(|_| "(no title)".into());
            println!("\nWindow {}: {}", i, title);

            // Print direct children
            if let Ok(children) = window.children() {
                for child in &children {
                    let role = child.role().unwrap_or_default();
                    let label = child.title().unwrap_or_default();
                    println!("  {} {}", role, label);
                }
            }
        }
    }
}

Save this as src/main.rs, run cargo run, and you will see the UI tree of whatever app is in the foreground. This is the foundation for building screen readers, UI test harnesses, and desktop automation agents in Rust.

Wrapping Up

The accessibility crate brings macOS AXUIElement into Rust with minimal overhead and safe abstractions. For high-frequency accessibility tree reads (desktop agents, automated testing, assistive technology), Rust's ownership model and zero-cost FFI make it a strong choice over scripting languages. Start with AXUIElement::system_wide(), walk the tree, and build from there.

Fazm is an open source macOS AI agent that uses the accessibility tree for screen understanding. Open source on GitHub.