[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-tool-retrieval-completeness-multitool-testbed":3,"topics-all":38,"news-related-e8f79d4f-9b2e-465a-94c1-cc725f09bc19":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"e8f79d4f-9b2e-465a-94c1-cc725f09bc19","多工具检索:召回率翻倍,完整率几乎不动","论文把六种工具检索结构放进两个真需要多工具的测试台,发现召回率翻倍时完整率几乎不动:综合分被单工具查询撑着,排序还会翻转。代价也分成两笔:标注过的请求换零查询成本,或每次查询几十次大模型调用换五分钟延迟。下游差距是 21 比 39。","六个工具检索结构放在同一张考卷上跑,跑出一个容易被忽略的结论:召回率翻倍的时候,完整率可能几乎不动。\n\n## 一份被平均值掩盖的评测口径\n\n论文的切入点是工具检索里一个很少被拆开看的指标:completeness@k——请求需要的整套工具,是不是全在检索结果的第 k 位以内。单工具请求里它退化成命中率;只有当请求需要多个工具时,它才真正开始测量检索器的水平。\n\n作者先在一份常用基准上做了分层。ToolRet 一共 7,961 条查询,其中 55% 只指向一个目标工具,单目标查询的 completeness@10 落在 0.58 到 0.67,而需要四个以上目标的查询只有 0.08 到 0.17。bge-base 的综合分 0.507,就是单目标 0.672 和多目标 0.303 的加权平均,一个被查询难度撑起来的数字。\n\n更要命的是排序会翻转。论文自己的评测里,单目标查询上 bge-base 最好或并列最好,但两个目标时,COLT 的 stage 1 检索器比它高 0.077 到 0.122,查询分解比它高 0.006 到 0.041;四个目标以上,分解反超 0.013 到 0.064。另一组独立研究在完全不同的注册表上得到同方向的交叉点:检索优先在单工具查询上占优,规划优先在多工具查询上占优,且在 k=1 到 20 的每个预算、1k 到 47k 个 API 的规模上都成立。\n\n## 换到多工具考场上,完成率掉到十分之一\n\n论文随后把战场换到两个真正需要多个工具的测试台。AppWorld 用 In-N-Out 的专家参数级依赖标注,345 条官方测试查询(129 条 test-normal、216 条 test-challenge)跑在 456 个 API 上,每条请求需要 2 到 12 个 API,中位数 6 个,其中 258 条单链条、87 条带依赖。测试台二 MCP-Bench 是 103 个任务、28 个服务器上的 257 个工具,其中 97 个任务明确需要多个工具。\n\nAppWorld 上的 C@16 很难看:BM25、MCP-Zero、HYSET 三个都是 0,bge-base 只有 0.020,查询分解 0.035,e5-base 0.064,依赖图 GoS 0.096。只有两个结构过线,COLT 0.328 和 AnyTool 0.301,而两者差值的置信区间 [-0.081, 0.029] 跨过零,分不出高下。MCP-Bench 上 COLT 以 0.583 领先(recall 0.887),AnyTool 0.398,查询分解 0.359。\n\n按服务器数量拆开,COLT 在单服务器任务上 0.782、两服务器 0.467、三服务器 0.167;三服务器任务里除 AnyTool 的 0.278 之外没人能完成任何一个。\n\n## 召回率和完整率根本不是一回事\n\n最能说明问题的一组数字在论文的第四段观察里:查询分解把 AppWorld 的 recall@16 从 bge-base 的 0.190 抬到 0.417,翻了一倍多,completeness@16 却只从 0.020 挪到 0.035。召回率说的是\"大部分工具找到了\",完整率问的是\"一整套齐了吗\"。MCP-Bench 上 AnyTool 和分解的 recall@16 是 0.783 和 0.776,看着已经追上 COLT 的 0.887,完整率却只有 0.398 和 0.359,离 COLT 的 0.583 还差一截。只盯着 recall 调检索器,调的是个跟能不能干活关系不大的指标。\n\n## 两种失败,两种修法\n\n论文把每一条不完整的检索拆开归因,两个领先结构的病灶正好相反。COLT 丢掉的 API 里 58%(404\u002F698)其实排在自己的第 17 到 50 名之间,MCP-Bench 上这个比例是 90%,属于排序深度不够,加大预算或加重排器就能捞回来。AnyTool 是另一种丢法:538 个缺失 API 里有 317 个属于\"没认出来的前置依赖\",即找到了请求点名的 API,却没把喂给它输入的那些一起收进来,修法是沿生产者-消费者关系补齐。平面检索和图检索则大多在检索起点就丢了目标。\n\n代价也是两笔完全不同的账。COLT 查询时一次大模型调用都不需要,0 token、0 延迟,代价是事前要在同一个工具池上准备标注过的请求,AppWorld 上是 90 条官方训练查询,MCP-Bench 上每折 75 条;AnyTool 不需要训练数据,但每次查询中位数 31 到 38 次大模型调用、99,739 token、307 到 311 秒墙钟时间,差不多五分钟一条。\n\n## 下游那一关:21 比 39\n\n最有说服力的是论文自己做的下游检查。用官方 AppWorld 的 ReAct 智能体跑 60 个任务,一组给标准答案 API,一组只给 COLT 的前 16 个:前者成功 46 次,后者 23 次,配对检验 p 值 5.6×10⁻⁶。论文自己的评测里,COLT 前 16 个凑齐的 21 个任务次次成功,标准答案在同样这批里成功 20 次;而 COLT 没凑齐的 39 个任务里,COLT 只成功 2 次,标准答案成功 26 次。缺工具时智能体会反复撞墙:步数中位数从 19 涨到 32,30 个回合撞上未暴露 API 的报错(标准答案组 8 次),prompt token 从 7.4M 涨到 15.1M。论文也诚实标注了这个比较是观察性的,凑齐的任务可能本身更容易,但同一批残缺任务换成标准答案能成功三分之二,差距不能全用任务难度解释。\n\n## 论文自己划的边界\n\n作者自己写明了两条限制:所有依赖大模型的环节都跑在 DeepSeek-V3.2 上,换成别的模型,依赖大模型能力的那条成本账可能重排(他引用的另一项数据显示,把网关模型从 GPT-5.4 换成 GPT-5.1,分层网关的任务成功率平均掉了 37.6 个点,而扁平化网关只掉 11.9 个点);另外所有稠密检索用的是 base 尺寸的编码器,所以那 10% 的天花板描述的是这个尺寸的编码器。\n\n所以\"所以呢\":给智能体接一整套工具,评测口径就该从 recall@k 换成 completeness@k,并按请求需要几个工具分层看,不然综合分会被单工具查询撑着走。资源账也要算清:工具池能配套标注请求,就用集合级检索器把查询时成本压到零;没有标注数据,分层导航能补上能力,但要准备好每条查询五分钟的延迟预算。要盯的不是\"找回来多少\",而是\"凑齐没有\"——下游数字是 21 比 39。\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.41905","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"360720c5-5b75-49e1-85e5-a7969841bb6d","en","Tool Retrieval: Recall Doubles, Completeness Barely Moves","Six retrieval structures: recall doubles but completeness barely moves, and rankings flip. Complete sets won 21 of 21 tasks, incomplete ones 2 of 39.","Six tool-retrieval structures were put on the same test paper, and the result is easy to miss: recall can double while completeness barely moves.\n\n## A metric buried under an average\n\nThe paper starts from a measure in tool retrieval that is rarely taken apart: completeness@k — whether the entire set of tools a request needs appears within the top k of the retrieval list. On single-tool requests the measure collapses into a hit rate; it only starts to measure the retriever when a request needs several tools.\n\nThe authors first stratify a standard benchmark. ToolRet has 7,961 queries, and 55% of them point at a single target tool. Split by target count the gap is immediate: completeness@10 lands between 0.58 and 0.67 on single-target queries, but falls to just 0.08 to 0.17 on queries needing four or more targets. bge-base's aggregate score of 0.507 is the weighted average of 0.672 on single-target and 0.303 on multi-target queries — a number propped up by the difficulty distribution of the query set.\n\nRankings also flip. In the paper's own evaluation, bge-base is best or tied for best on single-target queries, but at two targets the COLT stage 1 retriever beats it by 0.077 to 0.122 and query decomposition by 0.006 to 0.041; at four or more targets, decomposition is ahead by 0.013 to 0.064. A separate independent study finds a crossover in the same direction on a completely different registry: retrieve-then-plan designs win on single-tool queries, plan-then-retrieve designs win on multi-tool ones, and the crossover holds at every budget from k=1 to 20 and across registry sizes from 1k to 47k APIs.\n\n## On a multi-tool testbed, completion drops to a tenth\n\nThe paper then moves the fight to two testbeds that genuinely need several tools. AppWorld uses the expert parameter-level dependency labels from In-N-Out: 345 official test queries (129 test-normal, 216 test-challenge) run over 456 APIs, each request needing between 2 and 12 APIs with a median of 6, of which 258 are single chains and 87 carry dependencies. The second testbed, MCP-Bench, is 103 tasks over 257 tools on 28 servers, and 97 of those tasks explicitly need several tools.\n\nC@16 on AppWorld is grim: BM25, MCP-Zero and HYSET all sit at 0, bge-base reaches only 0.020, query decomposition 0.035, e5-base 0.064 and the dependency graph GoS 0.096. Only two structures clear the bar — COLT at 0.328 and AnyTool at 0.301 — and the confidence interval on the gap between them, [-0.081, 0.029], crosses zero, so they are indistinguishable. On MCP-Bench, COLT leads with 0.583 (recall 0.887), ahead of AnyTool at 0.398 and decomposition at 0.359.\n\nBroken down by server count, COLT scores 0.782 on single-server tasks, 0.467 on two-server and 0.167 on three-server. Among three-server tasks, nobody completes a single one except AnyTool at 0.278.\n\n## Recall and completeness are not the same thing\n\nThe most telling numbers sit in the paper's fourth observation: query decomposition lifts AppWorld recall@16 from bge-base's 0.190 to 0.417, more than doubling it, while completeness@16 moves only from 0.020 to 0.035. Recall says \"most tools were found\"; completeness asks \"is the whole set there?\" On MCP-Bench, AnyTool and decomposition post recall@16 of 0.783 and 0.776, apparently catching COLT's 0.887, yet completeness is only 0.398 and 0.359, still short of COLT's 0.583. Tuning a retriever on recall alone means tuning a metric that has little to do with whether the agent gets the job done.\n\n## Two failure modes, two different repairs\n\nThe paper decomposes every incomplete retrieval into causes, and the two leading structures fail in opposite ways. Of the APIs COLT loses, 58% (404 of 698) are actually sitting at ranks 17 to 50 of its own list — 90% of them on MCP-Bench — so it is a ranking-depth problem that a larger budget or a reranker could recover. AnyTool loses them differently: of 538 missing APIs, 317 are \"unrecognized prerequisites\", meaning it found the API the request names but never collected the APIs feeding its inputs, which points to completing the set along producer-consumer dependencies. Flat and graph retrievers mostly lose the target itself at the initial retrieval step.\n\nThe bills differ too. COLT needs no LLM call at query time, 0 tokens and 0 latency, but demands labelled queries over the same tool pool up front — 90 official training queries on AppWorld, 75 per fold on MCP-Bench. AnyTool needs no training data, but spends a median of 31 to 38 LLM calls per query, 99,739 tokens and 307 to 311 seconds of wall-clock time, roughly five minutes per query.\n\n## The downstream check: 21 against 39\n\nThe most persuasive piece is the downstream check the paper runs itself. The official AppWorld ReAct agent runs 60 tasks, one group given the gold APIs and one given only COLT's top 16: the former succeeds 46 times, the latter 23, with a paired test p-value of 5.6×10⁻⁶. In the paper's own evaluation, all 21 tasks where COLT's top 16 was complete succeeded every time, with the gold set succeeding on 20 of the same tasks; among the 39 tasks COLT failed to complete, COLT succeeded twice and the gold set succeeded 26 times. An agent short of tools keeps hitting walls: median steps rise from 19 to 32, 30 episodes run into an unexposed-API error (8 under the gold set), and prompt tokens go from 7.4M to 15.1M. The paper honestly flags the comparison as observational — complete tasks may simply be easier — but the same incomplete tasks succeed two thirds of the time under the gold set, so difficulty alone does not explain the gap.\n\n## The limits the paper sets for itself\n\nThe author states two limitations. Every LLM-dependent step runs on DeepSeek-V3.2, so swapping in a different model could reshuffle that cost line (another study he cites shows that moving the gateway model from GPT-5.4 to GPT-5.1 drops hierarchical gateway task success by 37.6 points on average, against 11.9 points for flattened gateways). And every dense retriever uses a base-size encoder, so that 10% ceiling describes encoders of that size.\n\nSo what does it mean in practice: if you are wiring a whole set of tools into an agent, the evaluation metric should move from recall@k to completeness@k, and it should be stratified by how many tools a request needs — otherwise the aggregate score gets propped up by single-tool queries. The resource bill matters too. If you can pair the tool pool with labelled requests, use a set-level retriever and push query-time cost to zero; without labelled data, hierarchical navigation can make up the capability gap, but budget for roughly five minutes of latency per query. What to watch is not \"how many came back\" but \"is the set complete\" — the paper's own downstream number is 21 against 39.\n","tool-retrieval-completeness-multitool-testbed","2026-11-02T11:15:00Z","2026-11-02T11:14:37.225072Z","2026-11-02T11:14:37.225094Z",true,"agent",325,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"43903d4a-7891-411b-9ea6-80708f7f8309","让大模型改权重:42道题测出能力边界","weightbench-llm-weight-space-programming","2026-11-16T07:20:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"c368ad9f-9308-4a0b-8f5c-3ae4601b48b9","D2K-Bench: 专家设计把 LLM 写 GPU 核提速 33.9%","d2k-bench-llm-gpu-kernel-design-guidance","2026-10-07T03:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"fe5546f3-09e4-42a3-bfde-6bbd0e8d474c","外星世界实测:探索 4 轮 87.6%,垫底 12.9%","explorationbench-alien-worlds","2026-09-28T21:07:30+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"3c6fcf46-f5bb-4136-931c-69cd64216e12","Skill-Use 基准揭示 Agent 短板：会做任务，不等于会用 Skill","skill-use-agent-harness-benchmark","2026-08-06T08:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"74de194b-9e2c-45ab-aa13-12fe210e66ba","HiGram 给 Agent 记忆加上“路径定位”：先找证据，再改记忆","higram-agent-memory-path-localization","2026-08-05T09:32:43+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"648ee35f-321e-4cc7-8e88-9a9d3ebae033","小模型每晚自学:一年后从 52 分掉到 5 分","on-device-llm-self-training-judge-failure","2026-11-19T03:30:00+00:00"]