Vercel Labs' agent-desktop Reaches 16,500 Stars With Three Releases in Six Days
Software / news
Vercel Labs' agent-desktop Reaches 16,500 Stars With Three Releases in Six Days
The Apache-2.0 CLI lets a coding agent drive desktop apps through the accessibility tree instead of screenshots. The README has no sandbox, and one Linux test found refs that did not save.
Vercel Labs' agent-desktop, a Rust command-line tool that lets an AI agent operate desktop applications, had 16,561 stars and 1,158 forks on GitHub on November 5. The repository was created on February 19, 2026. It shipped v1.0.2 on October 30, v1.1.0 on November 3 and v1.2.0 on November 5.
The README describes a loop the agent runs itself. The tool prints the operating system's accessibility tree as JSON, gives every actionable element a ref, and the agent clicks or types against a ref, then reads the tree again to check. In the README's words, "The loop lives in your agent; agent-desktop has no model inside."
How agent-desktop reads a screen without looking at it
Most computer-use tools send a screenshot to a model and get coordinates back. agent-desktop skips the picture where the OS exposes the control. It uses the AX API on macOS 13 and later, UI Automation on Windows 10 version 1809 and later, and AT-SPI on Ubuntu 24.04 and later.
The README lists 60 command names, 56 of them operational. Four, key-down, key-up, mouse-down and mouse-up, refuse to run until a daemon can release held input. Ref actions are headless by default and do not move the pointer or take focus unless --headed is passed.
Chris Tate introduced the project on October 31 as "agent-browser, for your whole computer", according to a deep dive by the developer Flavio on flaviocopes.com. That post is the only independent write-up The Terminal found, and its Linux transcripts come from a test of 1.0.2 on Ubuntu 24.04.5 with XFCE on X11, run on November 1.
- Full snapshot271 refs
- Skeleton snapshot79 refs
Source: Flavio, flaviocopes.com deep dive on agent-desktop, Ubuntu 24.04.5 test of 1.0.2, accessed 2026-11-05
What broke in the Linux test
The Linux run is where the friction shows. list-windows returned empty data for a window that was on screen, so snapshot failed with WINDOW_NOT_FOUND until the app was launched with --activate.
A full Mousepad snapshot came back with 271 refs, while a skeleton snapshot had 79. Dense apps, the README warns, need the skeleton map first and a drill into one region after.
Then the sticking point: set-value changed the unsaved text buffer, but the file on disk kept its original contents. Closing without --force failed on a pending Save Changes dialog. Two GTK clicks less than about 250 ms apart on one button acted once.
For a person scripting a form fill, that means "the field shows the new text" and "the data was saved" are different assertions, and the agent has to check the second one.
Three releases, one breaking
| Version | Date | What changed |
|---|---|---|
| 1.0.2 | October 30 | Release tested on Linux in the deep dive |
| 1.1.0 | November 3 | Breaking: stricter refs on Windows and Linux, macOS command host removed |
| 1.2.0 | November 5 | agent-desktop mcp server added, --stdin for text input |
Under 1.1.0 an old ref, or a ref that matches one of several identical controls, returns AMBIGUOUS_TARGET. Refs take the form @<snapshot_id>:eN, and a bare @e8 is rejected. A UI that changed since the snapshot returns STALE_REF. The README says to retry only when error.disposition.retry is safe.
The 1.2.0 MCP server exposes each leaf command as a tool, for example desktop_snapshot. The deep dive measured the native binary at about 21 ms of link time per call, against about 146 ms when install scripts are blocked and the Node launcher runs instead.
What the README says about safety
The README warns that snapshots, clipboard reads, screenshots, command output and debug artifacts "can contain private desktop content". Batch JSON text appears in process arguments, so secrets belong in separate commands using --stdin.
The deep dive adds that the project has no documented sandbox and does not discuss prompt injection from text on screen. Its author said they would run the tool in a virtual machine with test data, not on a daily-use machine. On macOS the tool needs the Accessibility permission, and on Windows the binary is unsigned, so SmartScreen or antivirus may flag it.
It is a different bet from the skill collections The Terminal has covered, such as Superpowers, and from agent-focused editors like JetBrains' Thinkrail: those shape what an agent knows, while this decides what it can touch.
Version 1.2.0, released on November 5, is the first to ship the MCP server, according to the deep dive.
Sources
More in Software
- 01Siri AI's Developer Preview Excludes TestFlight and the App Store, With Submissions Not Opening Until SpringApple's November 3 post opens App Intents integration to every developer on the 27.3 betas, but only development builds can use it, and no usage limits or fees are published.
- 02GitHub Copilot Reaches Xcode 27.3 Beta and JetBrains Air in a Day Apart, and Air Admins Cannot Cap SpendingThe November 3 and 4 changelog entries describe two different integrations: a public preview that needs macOS Tahoe 26.6, and an alpha-linked plugin with gaps in governance.
- 03Dependabot Now Bumps APM Agent Manifests, Version Updates Only, While APM's Scanner Misses Plain-Text Prompt InjectionGitHub's November 3 changelog covers git dependencies in apm.yml; Microsoft's APM docs say the package manager does not sign packages.
- 04Apple's Taiwan App Store Fee Falls to 26% While Outside Payment Links Stay Banned and the FTC Probe Is Only SuspendedThe 30% to 26% cut took effect October 31 under Article 28 of Taiwan's Fair Trade Act, and the regulator can reopen the case.