The way most computer-use agents work today is painfully familiar: take a screenshot of the desktop, hand it to a vision model, ask for pixel coordinates, click, take another screenshot to confirm. Anthropic's own documentation prices that loop out loud — a long agent run accumulates roughly 1,000 to 1,800 input tokens per screenshot, and merely declaring the computer toolset adds about 4,500 input tokens to every request. The screenshot is simultaneously the model's eyes and the single largest line item on the bill.

There is another route, one that never looks at a pixel at all: read the operating system's accessibility tree. agent-desktop, a project in Vercel Labs' GitHub organisation, takes that route. It reads the AX API on macOS, UI Automation on Windows and AT-SPI on Linux, and turns every interactive element into a JSON node carrying a ref. What reaches the model is not a picture. It is a tree.

The point of 1.1.0 is not features. It is refusal.

On 3 November, agent-desktop shipped v1.1.0. The changelog lists plenty of additions: macOS users can now observe Dock and menu-bar status items, background apps with no window can be observed through their process, and type gained an --interval-ms flag for paced keystrokes. But the entries actually marked as BREAKING CHANGES point somewhere else entirely.

The sharpest one: the C ABI major version is now 7, and the two "exact" lookup entrypoints — ad_resolve_element_exact and ad_execute_ref_action_exact_with_policy — are removed outright, always refusing with ACTION_NOT_SUPPORTED. The reason given is precise: a C caller cannot supply the capture proof that a ref now resolves by, so act through a snapshot ref instead.

Translated into plain language: the tool has decided it will no longer guess at which element you meant.

The response envelope moves to 2.6 in the same breath. On Windows and Linux, a ref that points at one of several identical controls, a ref from a window the snapshot could not read whole, or a ref from an earlier snapshot now returns AMBIGUOUS_TARGET carrying details.kind: identity_not_unique, where some of those cases used to act. And a snapshot that could not read its window whole to prove its refs now says so plainly, via data.identity_unproven: true.

The token arithmetic: 273 against 3,811

Saving tokens is the most legible argument for this design. The project's own figures: in Apple Maps, a single look drops from 3,811 tokens to 273; a simulated 52-step task falls from 198k tokens to 31k; and on dense Electron apps such as Slack, VS Code and Notion, skeleton traversal reportedly saves 78% to 96% versus dumping the full tree.

Those numbers deserve a discount. The coverage that exists labels them as self-reported, and the third-party write-up from techjacksolutions states plainly that no independent benchmark exists yet. What can be confirmed is that the direction has a real design rationale behind it: what you hand the model is structured JSON, not a bitmap. How much you actually save depends on your own application stack. The repository's locator-resolution benchmark runs against synthetic Chromium and Electron accessibility trees and reports p50 and p95 wall-clock latency, so it does not benchmark against a real product either.

What it cannot see

The boundaries are written into the official documentation, and they are unusually candid. agent-desktop does not automate web pages — that is the job of its sibling project, agent-browser; for Chromium and Electron apps it launches, agent-desktop opens a DevTools port that agent-browser connects to. The sharpest limit is structural: it reads accessibility trees and does not match pixels, so any interface that publishes no accessibility tree is invisible to it. Applications that draw their own canvas or ship custom-drawn widgets fall straight into that blind spot.

The project itself is young. The repository was created in February 2026 and began life as a personal project; it moved into the vercel-labs organisation in late October, with 1.0 shipping across late October and the public launch post going out on 30 October. It now carries 13,221 stars, the large majority of them arriving in the days right after the public announcement.

So what

1.1.0 deserves more attention than any of its new features.

The mainstream computer-use toolchain is chasing coverage: do the thing on whatever interface is in front of you, so click when you are unsure and deal with the consequences afterwards. agent-desktop 1.1 takes the other road — if you cannot be sure, do not click. Turning STALE_REF, AMBIGUOUS_TARGET and identity_unproven from "reason to retry" into "a formal answer inside the protocol" is an admission that in the agent era, an honest refusal is worth more than a blind click, because a wrong action is often not reversible while another look costs almost nothing.

One caveat is worth flagging: the strict semantics are currently enforced only on Windows and Linux. macOS behaves differently — identical controls each get their own ref, and resolution depends on whether the element's token is still alive. One tool running two different semantics across three operating systems is precisely the kind of seam that cross-platform projects tend to trip over.

So: does your desktop application's accessibility tree actually hold up?