[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-lucida-vlm-gizmoact-real-to-sim":3,"topics-all":38,"news-related-9a66407a-b79a-4a12-a4ef-b0d7018c8415":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"9a66407a-b79a-4a12-a4ef-b0d7018c8415","字节Seed新论文:VLM操作3D编辑器摆家具,把真实房间变成仿真场景","字节 Seed 团队新论文 Lucida 上线:把真实室内场景复刻为可单独编辑的仿真资产,核心 GizmoAct 让 VLM 像人一样多轮操作 3D 编辑器微调摆放,论文报告三项基准全面超越基线。","机器人仿真训练一直有个卡脖子的环节:怎么把一个真实房间,变成仿真器可以直接使用的数字场景。8 月 31 日挂上 arXiv 的论文 Lucida(arXiv:2608.30821)给出了一套新解法,Hugging Face 论文页把团队标注为 ByteDance Seed;上线第二天,它登上 HF Daily Papers 日榜第二,已获 63 个点赞。\n\n## 要解决的问题:三步流水线,步步都在假设理想输入\n\nComposable scene modeling(可组合场景建模)的目标是:输入一段杂乱的真实室内视频,输出一组完整、可单独编辑的 3D 物体资产,并按原样摆回场景——给机器人仿真和具身智能一份\"每个物体都能单独拿起来操作\"的场景复制品。\n\n论文指出,现有流水线把任务拆成三步:解析(parse)出物体实例、为每个实例生成(generate)资产、再把资产摆回去(place)。问题在于,每一步都假设上一步交付了理想输入——精确的实例几何、无遮挡的视角、与观测完全匹配的资产。而真实采集的视频,恰恰提供不了这些。\n\nLucida 的做法是保留三步顺序,但重新分配要求:每一步只消费真实采集\"可靠提供\"的信息,精度不要求在管线起点到位,而是推到末端闭环达成。\n\n## 核心亮点:GizmoAct,让 VLM 像人一样拖 gizmo\n\n前两步是扎实的工程整合:把视频解析成场景图,节点携带每个实例的多视角证据(参考视角、掩码、局部点云、3D 框、指代线索);再从证据合成无遮挡的物体图像,抬升成完整、可编辑的 3D 资产。\n\n真正有意思的是第三步的 GizmoAct。按项目页描述,这是一个 VLM 策略,把\"摆放\"变成多轮 GUI 交互:VLM 面向 3D 编辑器,每轮接收一组渲染观测(目标提示、相机证据、点云叠加、辅助视角、局部轴正交视图),在物体局部坐标系里发出一条可执行的位姿编辑指令——本质上是拖动 3D 软件里那个 gizmo 操纵柄——然后自己判断何时对齐、何时停止。\n\n翻译成人话:过去靠几何优化硬算位姿,现在是让视觉语言模型像 3D 美术一样——看一眼渲染结果,拖一下手柄,再看一眼,直到摆对。项目页的鲁棒性实验显示,同一个 GizmoAct 策略分别从 Boxer、Any6D、SAM 3D 三种不同来源的初始位姿出发(最多 4 视角、12 步精修),都能收敛到对齐状态。\n\n## 成绩单:三项基准,论文口径全面领先\n\n按论文摘要与项目页报告的数据(作者口径):\n\n- 场景级 3D 目标检测:R2S-Scene 上 mAP 较 Boxer 提升 69%;\n- 物体位姿估计:CA-1M 上 ADD-SB@0.05 从 57.8% 提至 83.4%;R2S-Object 上 GizmoAct 单视角 88.0%、最多 4 视角 92.0%,对比 SAM 3D 的 61.6%;\n- 场景重建:R2S-Scene 上 F-Score 从 SAM 3D 的 0.794 升至 0.924,Total CD 从 0.022 降至 0.010。\n\n## 值得聊的点:GUI 是个被低估的通用接口\n\nGizmoAct 最值得注意的不是分数,而是接口选择。Agent 社区这一年反复验证过\"让模型操作现成工具\"这条路径——浏览器、桌面应用、终端。Lucida 把同一套循环搬进 3D:不新造位姿回归网络,而是复用\"VLM 看 UI、发指令、收观测\"的通用 agent 循环,只是操作对象从按钮换成了 gizmo。\n\n这条路的好处是可解释、可插拔:3D 编辑器是成熟工具,渲染反馈是现成的误差信号,\"对没对齐\"的判断直接交给 VLM 的视觉能力。对做场景数据生成和 sim2real 的团队来说,这比端到端黑盒更容易调试,还能随 VLM 底座升级免费变强。\n\n当然要泼点冷水:上述基准数字全部来自作者报告,目前没有独立第三方复现;截至发稿,项目页也未挂出代码或权重链接,只有交互演示与 BibTeX。\n\n所以呢——如果你在做具身智能或仿真数据生产,这篇的启示是:别急着为\"空间智能\"专门造模型,先把\"VLM 加成熟工具的 GUI 接口\"这条便宜路径跑通。当 3D 操作也被收编进 agent 的 GUI 范式,空间智能可能只是通用 agent 能力的一层应用皮肤。\n\n参考:论文 arXiv:2608.30821(https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30821);项目页 https:\u002F\u002Flucida-r2s.github.io\u002F","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30821","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"6ad31a14-c0da-42df-81fd-564281f768db","agentic-ai",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",{"id":19,"name":20,"slug":20,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":22,"name":23,"slug":23,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"c6a563e6-362f-4bd0-a16d-c515b5364a08","en","VLM Learns to Place 3D Assets Like a 3D Artist: Lucida Turns Real Rooms Into Sim Scenes","ByteDance Seed-tagged paper Lucida turns room captures into editable 3D assets via GizmoAct, a VLM that drags gizmos in a 3D editor closed loop.","Robot simulation has a long-standing bottleneck: turning a real room into a digital scene a simulator can actually use. On August 31, a paper called Lucida (arXiv:2608.30821) landed on arXiv with a new answer. The Hugging Face paper page lists the team's affiliation as ByteDance Seed, and within a day the paper climbed to #2 on the HF Daily Papers board with 63 upvotes.\n\n## The problem: a three-step pipeline that assumes ideal inputs\n\nComposable scene modeling aims to take a cluttered indoor video capture and output a set of complete, individually editable 3D object assets, arranged exactly as observed — a simulation-ready replica of a real environment for robot simulation and embodied AI, where every object can be manipulated on its own.\n\nThe paper argues existing pipelines decompose this into three steps — parse observations into instances, generate an asset for each, then place each asset back — but every step presumes an input that cluttered captures rarely provide: accurate instance geometry, unoccluded views, and assets that accurately match observations.\n\nLucida keeps the order but redistributes the requirements: each step consumes only what a real capture reliably provides, and precision is reached at the end of the pipeline rather than demanded at its start.\n\n## The core idea: GizmoAct, a VLM that drags gizmos like a 3D artist\n\nThe first two stages are solid engineering: parse the video into a scene graph whose nodes carry per-instance multi-view evidence (reference views, masks, partial point clouds, 3D boxes, referring cues), then synthesize an occlusion-free object image from that evidence and lift it into a complete, editable 3D asset.\n\nThe interesting part is stage three's GizmoAct. Per the project page, it is a VLM policy that recasts placement as multi-turn GUI interaction: facing a 3D editor, the VLM receives a set of rendered observations each turn (target cue, camera evidence, point-cloud overlays, auxiliary views, local-axis orthographic views), issues one executable pose edit in the object's local frame — essentially dragging the gizmo handle in 3D software — and decides for itself when alignment is reached and it should stop.\n\nIn plain terms: instead of solving pose with geometric optimization, a vision-language model works like a 3D artist — glance at the render, drag the handle, glance again, repeat until it sits right. The project page's robustness section shows the same GizmoAct policy recovering from pose initializations produced by Boxer, Any6D, and SAM 3D (up to four views, 12 refinement steps).\n\n## The scorecard: three benchmarks, all author-reported\n\nPer the abstract and project page (author-reported numbers):\n\n- Scene-level 3D object detection: mAP improves over Boxer by 69% on R2S-Scene;\n- Object pose estimation: ADD-SB@0.05 rises from 57.8% to 83.4% on CA-1M; on R2S-Object, GizmoAct scores 88.0% single-view and 92.0% with up to four views, versus 61.6% for SAM 3D;\n- Scene reconstruction: scene F-Score on R2S-Scene goes from 0.794 (SAM 3D) to 0.924, with Total CD dropping from 0.022 to 0.010.\n\n## Why it matters: GUI is an underrated universal interface\n\nThe most notable thing about GizmoAct is not the scores but the choice of interface. Over the past year the agent community has repeatedly validated the \"let models operate existing tools\" path — browsers, desktop apps, terminals. Lucida ports that same loop into 3D: instead of training a new pose-regression network, it reuses the generic agent cycle of \"VLM looks at UI, issues an edit, collects observations,\" swapping buttons for gizmos.\n\nThe payoff is interpretability and pluggability. A 3D editor is a mature tool, rendered feedback is a ready-made error signal, and the \"is it aligned?\" judgment is delegated to the VLM's own vision. For teams building scene data generation or sim2real pipelines, this is easier to debug than an end-to-end black box — and it gets stronger for free as VLM backbones improve.\n\nSome cold water: all benchmark numbers above are author-reported with no independent replication yet, and as of writing the project page offers no code or weights release — only an interactive demo and BibTeX.\n\nSo what? If you work on embodied AI or simulation data production, the lesson is: before building a bespoke \"spatial intelligence\" model, try running the cheap path of \"VLM plus a GUI interface to a mature tool.\" Once 3D manipulation is absorbed into the agent GUI paradigm, spatial intelligence may turn out to be just one application skin over general agent capability.\n\nReference: arXiv:2608.30821 (https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.30821); project page https:\u002F\u002Flucida-r2s.github.io\u002F","lucida-vlm-gizmoact-real-to-sim","2026-09-01T19:10:00Z","2026-09-01T19:08:35.779581Z","2026-09-01T19:08:35.779598Z",true,"agent",150,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"051084dc-3da0-451e-9c6b-a267d5b0e77f","给机器人技能装上门禁:EmbodiedSkills 预检+验证闭环,RoboTwin 50 任务冲到 86.2%","embodiedskills-vla-verify-loop","2026-09-08T17:10:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"fdf05035-e02e-4954-8beb-c697ecc7975a","Cohere Parse 发布:$1.5 每千页的文档解析模型,ParseBench 79.2 超 Mistral OCR 4","cohere-parse-document-parsing-model","2026-08-28T13:30:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"3c6fcf46-f5bb-4136-931c-69cd64216e12","Skill-Use 基准揭示 Agent 短板：会做任务，不等于会用 Skill","skill-use-agent-harness-benchmark","2026-08-06T08:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"74de194b-9e2c-45ab-aa13-12fe210e66ba","HiGram 给 Agent 记忆加上“路径定位”：先找证据，再改记忆","higram-agent-memory-path-localization","2026-08-05T09:32:43+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"c9ba6037-e8c9-4007-98e5-32af59d92839","百度一镜 WAIC 首发数字人视频播客方案，文心多模态能力再突破","baidu-yijing-waic-digital-podcast","2026-07-19T08:02:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"e79481c2-0f1c-4e8c-9ab5-608869a257e8","EvoQuality 开源：字节用「自投票 + GRPO」让 VLM 在零标注下学会图像质量评估","evoquality-bytedance-self-voting-grpo-iqa","2026-06-12T02:00:00+00:00"]