[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-robotok-retrieval-benchmark-reread":3,"topics-all":41,"news-related-58ed753e-ad6d-4aac-95f4-36bf217e169c":42},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":27,"news_slug":34,"published_at":35,"created_at":36,"modified_at":37,"is_published":38,"publish_type":39,"image_url":14,"view_count":40},"58ed753e-ad6d-4aac-95f4-36bf217e169c","把 10 万条人类视频变成机器人教材:RoboTok 检索 mAP 提升约 50 倍,hard 任务 79.3% 对 19.5%","莱斯大学 RobotPI Lab 联合 NVIDIA 发布 RoboTok:10 万条网络操作视频按 3D 手部轨迹建成可检索嵌入空间,mAP@20 从 0.0071 提到 0.3531,约 50 倍;VTDexManip 5\u002F6 任务最高、hard 3 任务全胜;真机结果未公布。","对机器人来说,学会一个新的操作动作从来不是难点,难点是演示数据从哪来。遥操作真机采一批演示贵到让账本爆炸,而现实任务的长尾永远等不起慢慢采集。莱斯大学 RobotPI Lab 与 NVIDIA 的论文 RoboTok(arXiv 2609.03199)换了个思路:人类早把各种操作上传到视频网站了,缺的只是一个靠谱的「检索器」。\n\n## 方法:比手部轨迹,不比画面\n\nRoboTok 把互联网操作视频转成 3D 手部轨迹,并表达在估计出的躯干参考系里——手的运动相对人自己的身体描述,相机视角、场景外观、人物遮挡随之被消掉,不同视频里的同类操作变成同一坐标系下可直接比较的向量。训练时用 DTW 度量轨迹相似度监督一个紧凑嵌入空间;查询时只做余弦相似度搜索,不必对全库逐对算 DTW,互联网级持续索引才可行。即使人已出画,模型也能凭两只手的轨迹估计出躯干坐标系。语料 10 万条片段:9 万条训练、1 万条留出做查询。\n\n## 检索表:0.3531 对 0.0071\n\n自建评测集上,RoboTok 的 mAP@20 为 0.3531,此前最强的 STRAP(ICLR 2025)是 0.0071,差约 50 倍;MRR@20 为 0.8576 对 0.0824。在带传感器级 3D 手部标注的 AssemblyHands 语料(831 条双手装配片段)上做域外测试,mAP@5 为 0.2614,约为 STRAP(0.1330)的两倍。按 DTW 成本看,RoboTok 检索结果 1.333,已接近完美检索的 1.145。\n\n## VTDexManip:hard 榜几乎一边倒\n\n下游才是重点。论文报告,RoboTok 检索的演示让策略在 VTDexManip 6 个原始任务中的 5 个、以及 3 个自改 hard 任务(手自由运动、去掉密集奖励)的全部任务上拿到最高成功率。hard 版 Lever Sliding:RoboTok 79.3%,最强基线 HAND(ICRA 2026)19.5%;hard 版 Faucet Screwing:44.8% 对 6.8%。策略输入只有本体感知和指尖力,不靠视觉。\n\n## 冷水:数字全在仿真里\n\n项目页真机结果一栏写的是 coming soon。mAP 0.35 距上界 1.0 仍远,「远超基线」不等于「够用」。开源仓库 RoboTok-Code(MIT 许可)不含那 10 万条视频库:数据构建代码依赖未随仓库发布的私有片段数据库,MANO\u002FSMPL-H 手部模型需另行注册获取——想复现「互联网级」全链路,先得自攒语料。\n\n## 所以呢\n\nLLM 时代「嵌入 + 向量检索」这套基础设施,正在平移进具身智能的数据层。当人类视频库本身能当机器人教材,瓶颈便从「采集」转向「检索质量与真机闭环」——这套数字能否在真机上站住,是论文的下一道题。\n\n参考:arXiv:2609.03199 \u002F rice-robotpi-lab.github.io\u002FRoboTok\u002F","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.03199v1","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21,24],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",{"id":22,"name":23,"slug":23,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":25,"name":26,"slug":26,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",[28],{"id":29,"lang":30,"title":31,"summary":32,"content":33},"97f1ae5a-9e95-46c3-a8ec-d17ec003398f","en","RoboTok: retrieval mAP up ~50x to 0.3531, hard tasks 79.3% vs 19.5%","RoboTok embeds 100K web videos via torso-relative 3D hand trajectories: mAP@20 0.3531 vs 0.0071 prior best; hard tasks 79.3% vs 19.5%. Sim-only.","For robots, learning a new manipulation skill is rarely the hard part — the hard part is where demonstration data comes from. Teleoperating a real robot to collect demonstrations blows up any budget, and the long tail of real-world tasks cannot wait. RoboTok (arXiv 2609.03199), from Rice University's RobotPI Lab with NVIDIA, flips the premise: humans have already uploaded the manipulations to video sites; what is missing is a reliable retriever.\n\n## Method: compare hand trajectories, not pixels\n\nRoboTok converts internet manipulation videos into 3D hand trajectories expressed in an estimated torso-relative reference frame. Because motion is described relative to the actor's own body, camera viewpoint, scene appearance, and occlusions cancel out, and similar manipulations across videos become directly comparable vectors. A compact embedding space is trained with DTW-based motion-similarity supervision; queries are cosine-similarity search over embeddings, with no pairwise DTW over the corpus — that is what makes internet-scale continual indexing feasible. Even when the actor is out of frame, the model estimates the torso frame from the two hand trajectories alone. The corpus holds 100,000 clips: 90,000 for training, 10,000 held out as evaluation queries.\n\n## Retrieval table: 0.3531 versus 0.0071\n\nOn the in-domain evaluation set, RoboTok reaches mAP@20 of 0.3531 against 0.0071 for the strongest prior method, STRAP (ICLR 2025) — roughly a 50x gap; MRR@20 is 0.8576 versus 0.0824. On AssemblyHands, an out-of-domain corpus of 831 two-hand assembly clips with sensor-grade 3D hand annotations, mAP@5 is 0.2614, about double STRAP's 0.1330. By DTW cost, RoboTok's retrieved results sit at 1.333, close to perfect retrieval's 1.145.\n\n## VTDexManip: hard mode is nearly one-sided\n\nDownstream is the point. Per the paper, demonstrations retrieved by RoboTok give policies the highest success rate on 5 of 6 original VTDexManip tasks and on all 3 modified hard tasks (free 3D hand motion, dense rewards removed). Hard-mode Lever Sliding: RoboTok 79.3% versus 19.5% for the best baseline, HAND (ICRA 2026); hard-mode Faucet Screwing: 44.8% against 6.8%. Policies use only proprioception and fingertip force — no vision.\n\n## Cold water: every number is in simulation\n\nThe real-world section of the project page reads \"coming soon.\" An mAP of 0.35 is still far from the 1.0 upper bound; \"far beyond baselines\" is not \"good enough.\" The open-source repo RoboTok-Code (MIT license) does not include the 100K-video corpus: dataset-building code depends on a private clip database that is not shipped, and the MANO\u002FSMPL-H hand models require separate registration — reproducing the \"internet-scale\" pipeline end to end means assembling your own corpus first.\n\n## So what\n\nThe embedding-plus-vector-search infrastructure of the LLM era is migrating into the data layer of embodied AI. When humanity's video libraries can serve as robot textbooks, the bottleneck moves from collection to retrieval quality and real-world closure. Whether these numbers survive contact with real robots is the paper's next question.\n\nRefs: arXiv:2609.03199 \u002F rice-robotpi-lab.github.io\u002FRoboTok\u002F","robotok-retrieval-benchmark-reread","2026-09-06T21:11:25Z","2026-09-06T21:12:18.800128Z","2026-09-06T21:12:18.800137Z",true,"agent",54,[],{"items":43},[44,49,54,59,64,69],{"id":45,"title":46,"news_slug":47,"published_at":48},"005557c5-8a3c-4d34-89bc-35d5351c4570","蒸馏只需要一条训练样本?清华实测:单条query覆盖71.5%训练状态,16条追平17k全量","one-shot-opd-single-query-distillation","2026-09-05T21:07:11+00:00",{"id":50,"title":51,"news_slug":52,"published_at":53},"7623f190-7071-4811-a6f1-32462a99b8d3","经验会过期:阿里云论文让自主后训练的有害授权率从 62.5% 降到 25%","bcit-conditional-experience-transfer-post-training","2026-09-05T17:11:11+00:00",{"id":55,"title":56,"news_slug":57,"published_at":58},"4a89fe5a-8703-49e5-b083-079cbda0fa2a","蒸馏也有副作用:中间训练期上KD,推理上涨、事实记忆反而变慢","switch-distillation-midtraining-kd","2026-09-02T17:10:00+00:00",{"id":60,"title":61,"news_slug":62,"published_at":63},"f9cf9f03-6aca-4d29-94d3-5c6acfeaf435","匿名模型 OX Alpha 短暂登顶 OpenRouter 编码榜:研究者推测底座指向智谱 GLM-5.x","ox-alpha-stealth-openrouter-glm-5-zhipu","2026-08-24T03:00:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"38fe9093-827f-43d1-8350-7cdd391cf1e3","北大 DataPrep-Bench 把 LLM 当数据准备工来打分：DAS 评估器把「训练价值」算成分布距离","pku-dataprep-bench-das","2026-07-27T22:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"d4e4cbdc-ddde-458d-9649-e53bce5ddbce","MLPerf Training v6.0 把 MoE 钉在牌桌中央：671B 训练首次纳入工业基准，FP4 路径开始分裂","mlperf-training-v6-moe-671b-fp4-split","2026-06-16T18:00:00+00:00"]