Vercel Labs' agent-desktop Reads Apps Through Accessibility Trees, and 63 Open Issues Show Where That Breaks

Vercel Labs' agent-desktop, a Rust command-line tool that lets an AI agent read and operate desktop applications on macOS, Windows and Linux, picked up 2,470 stars on GitHub trending on Nov. 5. The repository shows 15.3k stars, 1.1k forks, an Apache-2.0 licence and 63 open issues.

The README says it works from the operating system's accessibility tree rather than screenshots, and that the tool contains no model. The agent's own loop decides what to click.

How it sees an app

A snapshot command returns JSON describing each control, with a reference the agent can act on, such as a click or a text entry. The README names the three accessibility layers it uses.

Platform	Minimum	Accessibility API
macOS	13, Intel and Apple Silicon	AX API
Windows	10 1809, x64 and ARM64	UI Automation
Ubuntu	24.04, X11 or Wayland	AT-SPI

The README counts 60 command names, of which 56 operate. Four (key-down, key-up, mouse-down, mouse-up) refuse to run until a daemon can release held input. An mcp mode exposes each command as a tool over stdio, though issue 554 reports that the MCP stdio server does not accept an mcp argument.

What one Linux run found

The independent write-up is by Flavio on flaviocopes.com, updated Nov. 4. He ran version 1.0.2 on Ubuntu 24.04.5 with XFCE on X11, opened Mousepad, and took snapshots.

The full Mousepad snapshot held 271 references; a skeleton snapshot held 79. He measured about 21 ms per call with the native binary and about 146 ms through the Node launcher.

His most useful finding is a small one. set-value changed the text in Mousepad's buffer, but the file on disk did not change. "The file was not saved, though," he wrote. An agent that reports success after set-value has edited a window, not a document.

He also could not snapshot Thunar until it was launched with a window. He wrote that Wayland limits pointer input and window control, and that Windows binaries are unsigned, so SmartScreen may block them.

Where the issue tracker points

The newest open issues cluster on Windows and permissions, not on Linux.

• Issue 569: a snapshot of a higher-privilege window returns empty but reports complete.
• Issue 567: headless Windows UI Automation actions activate background windows.
• Issue 562: window enumeration fails inside the Codex sandbox but works in PowerShell.
• Issue 529: on Linux X11, press into Chrome's address bar crashes Chrome.
• Issue 534: a suggestion to guard the Escape key when a modal may discard work.

Issue 569 matters most. An empty snapshot marked complete tells an agent that nothing is there, and the agent will act on that.

What the docs do not say

The README warns that snapshots, clipboard reads, screenshots and debug artifacts "can contain private desktop content" and should stay out of public logs. It offers --stdin for type, set-value and clipboard-set so text stays out of process arguments, and warns that batch JSON text does appear in process arguments.

Flavio wrote that "agent-desktop does not document a sandbox of its own" and that the docs do not mention prompt injection from on-screen text. He recommends a virtual machine with test data and no logins, and wrote, "I would keep it off my everyday desktop." For scale on unattended agents, Hugging Face's own intrusion timeline counts one agent and 17,600 actions. Another trending agent add-on, claude-mem, hooks into six agent events; agent-desktop works at the other end of the loop, on the screen.

The tracker is the thing to watch: 63 open issues on Nov. 5, with permission and empty-snapshot bugs on top.