[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-agent-desktop-1-1-accessibility-tree-desktop-control":3,"topics-all":38,"news-related-05d3d178-c262-4f91-b977-0eac1ad91ac1":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"05d3d178-c262-4f91-b977-0eac1ad91ac1","不截图也能操控桌面:Vercel 的 agent-desktop 1.1","Vercel Labs 的 agent-desktop 放出 1.1.0:不截图,直接读 macOS\u002FWindows\u002FLinux 的无障碍树,把元素变成带 ref 的 JSON 节点,自报单次观察 token 从 3811 降到 273。重点是拒绝:身份不唯一就返回 AMBIGUOUS_TARGET,宁可不点也不猜。","现在的 computer use，大概率是这样跑的：给模型一张桌面截图，让它输出像素坐标，点一下，再截一张图确认。Anthropic 自己的文档把这条路的代价写得很直白——长链路里每张截图约 1,000–1,800 input token，光是声明 computer 工具集本身还要再加约 4,500 token（[Anthropic computer use 文档](https:\u002F\u002Fplatform.claude.com\u002Fdocs\u002Fen\u002Fagents-and-tools\u002Ftool-use\u002Fcomputer-use-tool)）。截图既是模型的\"眼睛\"，也是账单上最大的那一栏。\n\n另一条路是彻底不看像素：读操作系统的无障碍树。Vercel Labs 名下的 agent-desktop 走的就是这条——它读 macOS 的 AX API、Windows 的 UI Automation、Linux 的 AT-SPI，把每个能交互的元素变成一个带 ref 的 JSON 节点。模型拿到的不是图片，是一棵树。\n\n## 1.1.0 的重点不是功能，是拒绝\n\n11 月 3 日，agent-desktop 放出 1.1.0（[release notes](https:\u002F\u002Fgithub.com\u002Fvercel-labs\u002Fagent-desktop\u002Freleases\u002Ftag\u002Fv1.1.0)）。CHANGELOG 里新增功能不少：macOS 上能观察 Dock 和菜单栏状态项、能观测没有窗口的后台应用、`type` 支持按间隔逐键输入。但真正标成 BREAKING CHANGES 的几条，指向另一件事。\n\n最刺眼的一条：C ABI 大版本升到 7，而 `ad_resolve_element_exact` 和 `ad_execute_ref_action_exact_with_policy` 两个精确查找入口被直接废掉，一律返回 ACTION_NOT_SUPPORTED。理由写得很具体：C 调用方提供不了 ref 所依赖的 capture proof，请改走 snapshot ref。\n\n翻译成人话：这个工具拒绝再\"尽力猜一个元素\"了。\n\n响应信封同步升到 2.6。新规则是，在 Windows 和 Linux 上，ref 若指向多个长得一样的控件、来自没能完整读取的窗口、或者来自更早的快照，一律返回 AMBIGUOUS_TARGET 并带上 details.kind = identity_not_unique；连\"某个快照压根没读全窗口\"都不再藏着，用 data.identity_unproven: true 标出来。\n\n## 成本账：273 对 3,811\n\n省 token 是这条路线最直观的卖点。官方自报的数据是：在 Apple Maps 上，单次\"看\"的 token 从 3,811 降到 273；一个模拟的 52 步任务，总量从 198k 降到 31k；在 Slack、VS Code、Notion 这类密度较高的 Electron 应用上，走 skeleton traversal 省 78%–96%。\n\n这些数字得打折扣。多家转载都标注为作者自报，第三方技术站 techjacksolutions 也明确写了目前没有独立基准可对照。能确认的是方向有设计上的合理性——传给模型的是结构化 JSON，不是像素图；至于具体省多少，得在自己的应用栈上量。仓库里的 locator-resolution 基准跑的是合成 Chromium\u002FElectron 无障碍树，报 p50 与 p95 延迟，也不与真实产品对标。\n\n## 它看不见什么\n\n边界写在官方文档里，而且相当诚实。agent-desktop 不做网页自动化，那是兄弟项目 agent-browser 的活；它启动的 Chromium\u002FElectron 应用，则开 DevTools 端口让 agent-browser 接上。最要命的一条是：只读无障碍树、不做像素匹配，所以任何不发布无障碍树的界面，它就是完全看不见——自绘 canvas、自己画控件的应用，都可能落在盲区。\n\n项目本身也年轻。仓库 2026 年 2 月建立，最初是个人项目，10 月底转入 vercel-labs 组织，1.0 版本也在 10 月底才陆续发出，对外的发布帖则是 10 月 30 日。目前星标数 13,221，绝大多数集中在公开之后那几天。\n\n## 所以呢\n\n1.1.0 比任何新功能都更值得记住。\n\n主流 computer use 工具几乎都在追覆盖率——尽量在任意界面上把事做成，于是认不准就点，点错了再说。agent-desktop 1.1 选了另一条：认不准就不点。把 STALE_REF、AMBIGUOUS_TARGET、identity_unproven 从\"重试理由\"改写成\"协议里的正式答案\"，等于承认在 agent 时代，一次诚实的拒绝比一次盲目的点击更值钱——错误动作往往不可回滚，而重新观察几乎不要钱。\n\n值得留个心眼的是，这套严格语义目前只在 Windows 和 Linux 上强制执行；macOS 行为不同，同名控件各拿各的 ref，靠元素 token 是否存活来判断。同一个工具跨三种系统跑出两种语义，是这类跨平台项目最容易踩的坑。\n\n那你的桌面应用，无障碍树够用吗？\n","https:\u002F\u002Fgithub.com\u002Fvercel-labs\u002Fagent-desktop\u002Freleases\u002Ftag\u002Fv1.1.0","998df6db-96e6-4b8e-8be1-cfa00a6cd177",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"e82b2d09-81b2-43d1-977e-e018443b3c14","coding-agent",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"045c011e-e2bb-45ce-bdd6-0c927f8a3b87","token-efficiency",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"3b10829c-5b36-43b9-bd07-c2412614c9d6","en","agent-desktop 1.1: Desktop Control Without Screenshots","agent-desktop 1.1 reads accessibility trees, not screenshots, and refuses ambiguous refs with AMBIGUOUS_TARGET; the builder reports 273 tokens a look.","The way most computer-use agents work today is painfully familiar: take a screenshot of the desktop, hand it to a vision model, ask for pixel coordinates, click, take another screenshot to confirm. Anthropic's own documentation prices that loop out loud — a long agent run accumulates roughly 1,000 to 1,800 input tokens per screenshot, and merely declaring the computer toolset adds about 4,500 input tokens to every request. The screenshot is simultaneously the model's eyes and the single largest line item on the bill.\n\nThere is another route, one that never looks at a pixel at all: read the operating system's accessibility tree. agent-desktop, a project in Vercel Labs' GitHub organisation, takes that route. It reads the AX API on macOS, UI Automation on Windows and AT-SPI on Linux, and turns every interactive element into a JSON node carrying a ref. What reaches the model is not a picture. It is a tree.\n\n## The point of 1.1.0 is not features. It is refusal.\n\nOn 3 November, agent-desktop shipped v1.1.0. The changelog lists plenty of additions: macOS users can now observe Dock and menu-bar status items, background apps with no window can be observed through their process, and `type` gained an `--interval-ms` flag for paced keystrokes. But the entries actually marked as BREAKING CHANGES point somewhere else entirely.\n\nThe sharpest one: the C ABI major version is now 7, and the two \"exact\" lookup entrypoints — `ad_resolve_element_exact` and `ad_execute_ref_action_exact_with_policy` — are removed outright, always refusing with ACTION_NOT_SUPPORTED. The reason given is precise: a C caller cannot supply the capture proof that a ref now resolves by, so act through a snapshot ref instead.\n\nTranslated into plain language: the tool has decided it will no longer guess at which element you meant.\n\nThe response envelope moves to 2.6 in the same breath. On Windows and Linux, a ref that points at one of several identical controls, a ref from a window the snapshot could not read whole, or a ref from an earlier snapshot now returns AMBIGUOUS_TARGET carrying `details.kind: identity_not_unique`, where some of those cases used to act. And a snapshot that could not read its window whole to prove its refs now says so plainly, via `data.identity_unproven: true`.\n\n## The token arithmetic: 273 against 3,811\n\nSaving tokens is the most legible argument for this design. The project's own figures: in Apple Maps, a single look drops from 3,811 tokens to 273; a simulated 52-step task falls from 198k tokens to 31k; and on dense Electron apps such as Slack, VS Code and Notion, skeleton traversal reportedly saves 78% to 96% versus dumping the full tree.\n\nThose numbers deserve a discount. The coverage that exists labels them as self-reported, and the third-party write-up from techjacksolutions states plainly that no independent benchmark exists yet. What can be confirmed is that the direction has a real design rationale behind it: what you hand the model is structured JSON, not a bitmap. How much you actually save depends on your own application stack. The repository's locator-resolution benchmark runs against synthetic Chromium and Electron accessibility trees and reports p50 and p95 wall-clock latency, so it does not benchmark against a real product either.\n\n## What it cannot see\n\nThe boundaries are written into the official documentation, and they are unusually candid. agent-desktop does not automate web pages — that is the job of its sibling project, agent-browser; for Chromium and Electron apps it launches, agent-desktop opens a DevTools port that agent-browser connects to. The sharpest limit is structural: it reads accessibility trees and does not match pixels, so any interface that publishes no accessibility tree is invisible to it. Applications that draw their own canvas or ship custom-drawn widgets fall straight into that blind spot.\n\nThe project itself is young. The repository was created in February 2026 and began life as a personal project; it moved into the vercel-labs organisation in late October, with 1.0 shipping across late October and the public launch post going out on 30 October. It now carries 13,221 stars, the large majority of them arriving in the days right after the public announcement.\n\n## So what\n\n1.1.0 deserves more attention than any of its new features.\n\nThe mainstream computer-use toolchain is chasing coverage: do the thing on whatever interface is in front of you, so click when you are unsure and deal with the consequences afterwards. agent-desktop 1.1 takes the other road — if you cannot be sure, do not click. Turning STALE_REF, AMBIGUOUS_TARGET and identity_unproven from \"reason to retry\" into \"a formal answer inside the protocol\" is an admission that in the agent era, an honest refusal is worth more than a blind click, because a wrong action is often not reversible while another look costs almost nothing.\n\nOne caveat is worth flagging: the strict semantics are currently enforced only on Windows and Linux. macOS behaves differently — identical controls each get their own ref, and resolution depends on whether the element's token is still alive. One tool running two different semantics across three operating systems is precisely the kind of seam that cross-platform projects tend to trip over.\n\nSo: does your desktop application's accessibility tree actually hold up?\n","agent-desktop-1-1-accessibility-tree-desktop-control","2026-11-04T05:05:00Z","2026-11-04T05:14:12.297698Z","2026-11-04T05:14:12.297706Z",true,"agent",136,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"278dc1cc-ab13-4de3-9ed4-27f6732dda3c","五个27B组队靠账本编排追平Fable 5,复现只承认打平","gvs5h-ledger-orchestration-qwen-fable-rerun","2026-11-01T09:20:28+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"05fe5d2f-b83d-4aa9-9a68-5d0274b0d558","DeepSeek 开源 Super-agent:双层架构调度","deepseek-super-agent-router-worker-architecture","2026-11-01T07:31:50+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"c10ebc48-1efe-409e-bfc3-7f00c6e04768","Velum 不是单文件 C++:它是 DeepSeek-V4-Pro 和 Claude Code 的协作样本","velum-deepseek-claude-code-collab-sample","2026-09-28T07:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"c696208b-6535-4eb9-b1ed-2e4f835d2f88","NVIDIA SoL-Pi 把 coding agent 的 token 砍掉 44%,harness 开始变天","nvidia-sol-pi-harness-token-compression","2026-09-19T03:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"63c30bcd-3ffc-47c5-bd74-c2a9ed8f7c94","DeepSeek Harness 预览版开源:Agent 被拆成可插拔的插件栈,模型只负责想、Harness 负责做事","deepseek-harness-plugin-stack","2026-09-05T06:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"32b938b6-01a3-43c9-b040-14db6c5f57c6","NVIDIA 把 Agent 装进一个 Python 类:被忽略的 NOOA,一半 token 跑出 SWE-bench 82.2%","nvidia-nooa-python-agent-framework","2026-08-23T17:20:00+00:00"]