[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-omniassistbench-omni-llm-video-assistant":3,"news-related-d74a088d-e7f5-41cc-8c55-aedb10b8101d":38},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"d74a088d-e7f5-41cc-8c55-aedb10b8101d","Gemini-3-Pro 也只拿 66.4 分:南京大学开源全模态视频助手基准 OmniAssistBench","南京大学联合南开、滑铁卢大学开源 OmniAssistBench:将互联网视频逆向工程为 685 组多轮交互问答,评测全模态模型担任实时视频助手的水平。榜首 Gemini-3-Pro 仅 66.4\u002F100,开源最优 51.2;手势指令、上下文保持与适时沉默是三大短板。","全模态大模型的下一站在哪?各家把语音、视觉、文本塞进同一个模型之后,「实时视频助手」成了最受关注的落地形态:模型持续看着你的屏幕或周围环境,通过多轮对话引导你完成具体目标。但一个尴尬的问题是——怎么评测这种能力?静态的视频问答数据集回答不了这件事,因为模型一旦给出建议,用户接下来的动作就会改变,后续视频内容也随之变化,离线数据集根本无法预设这些分支。\n\n8 月 21 日提交到 arXiv 的 OmniAssistBench 给出了一套解法。这项工作由南京大学牵头,联合南开大学与滑铁卢大学完成,论文编号 arXiv:2608.21360([论文地址](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.21360)),代码与数据已在 GitHub 开源。\n\n## 逆向工程:把互联网视频切成「交互」\n\n研究团队的思路是把现成的互联网视频「逆向工程」成交互数据:先从视频中推导出合理的用户目标,再把视频切成多轮片段来模拟连续交互。为了解决「同一个目标可以有多条完成路径」的发散问题,他们给模型提供从源视频推导的先验知识,要求模型沿着完全相同的路线引导用户。\n\n整个数据集包含 685 组开放式问答,覆盖 7 大任务类型、16 个细粒度任务,视频题材横跨运动、烹饪、讲座、DIY 和脱口秀。标注管线分四个阶段,平均每个样本要花费约 4 小时专家工时,总投入按论文口径超过 1000 个专家小时。评测采用 LLM-as-a-Judge 流程,先打 0-5 分再归一化到 0-100。\n\n值得一提的是题型设计:所有用户问题不以文本形式输入,而是直接嵌进视频里——多数转成 TTS 语音,部分以字幕、手写画面或画中画手势呈现,模型只能看到视频本身。多轮模拟也讲究时序因果:当前轮的视频从上一轮结束处开始,用户提示嵌在片段末尾,尽量还原在线流式交互的因果结构。\n\n## 榜单:榜首 66.4 分,没有赢家\n\n评测覆盖 11 个模型,结果谈不上乐观。榜首的 Gemini-3-Pro 拿到 66.4\u002F100,第二名 Gemini-2.5-Pro 为 64.6,字节跳动的 Doubao-Seed-2.0-lite 以 57.3 排第三,小米 MiMo-V2-Omni 53.8、Qwen3.5-Omni-Plus 51.6 紧随其后。开源模型中表现最好的 Qwen3-Omni-Instruct(30B-A3B)只有 51.2 分,末端的开源模型 VITA-1.5 仅 24.6。\n\n更有意思的是团队自拍的 3 个真实案例(平均 15 轮交互,涵盖模拟会议、盲人导航辅助、多人手工进度跟踪):Gemini-2.5-Pro 总分第二,Real Cases 却只有 44.8,反而低于 Qwen3-Omni-Instruct 的 53.8;Gemini-3-Pro 在这一项拿到 68.0 保持第一。这说明旧旗舰在总榜上的分数,与真实长交互中的表现可以严重脱节。\n\n## 三块短板,块块指向下一代设计\n\n论文总结了当前全模态模型的三个关键瓶颈:\n\n- **视觉提示失灵**:模型难以把手势识别为用户指令,手势类提示的跟随普遍失效;\n- **上下文耗尽**:视频和音频的 token 消耗极快,缺少长期记忆机制的模型几分钟内就耗尽上下文窗口,随后忘记用户目标、给不出有效引导;\n- **延时响应失败**:模型分不清什么时候不该说话,经常输出与用户提示关系不大的视频描述,而不是在视觉证据不足时保持沉默。\n\n按团队的评分口径,40-60 分区间意味着「模型大致知道该干什么,但给不出准确且完整的回答」——榜单大部分模型正落在这个区间。\n\n## 所以呢\n\nOmniAssistBench 的价值不只是多一个排行榜。它把「视频理解」和「视频协作」划出了明确分界线:前者是被动看片答题,后者要求模型主动对齐视觉状态、用户目标与先验知识,还要拿捏回应时机。Real Cases 的分化尤其值得警惕——离线刷分与真实长交互之间的沟,比总榜数字呈现的更深。对想做 AI 视频助手产品的团队来说,上线前先在这个基准上跑一遍,比看发布会演示实在得多。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.21360","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"a9524a82-a7c5-4daa-bb4b-a7ee77bb0b94","gemini",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"3b1ef95a-8eb6-4da7-9b22-13ee6c729288","en","Gemini-3-Pro Scores Only 66.4 on OmniAssistBench","OmniAssistBench tests omni-LLMs as live video assistants via 685 multi-turn QA pairs. Gemini-3-Pro tops at 66.4\u002F100; best open model hits 51.2.","The next frontier for omni-modal large models is the real-time video assistant: a model that continuously watches the environment and guides the user toward a goal through multi-turn conversation. The awkward part has always been evaluation. Static video-QA datasets cannot simulate this setting, because the model's own responses dynamically change what the user does next — and therefore what the video contains.\n\nOmniAssistBench, submitted to arXiv on Aug 21 (arXiv:2608.21360, [paper](https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.21360)) by a Nanjing University-led team with Nankai University and the University of Waterloo, offers a way out of that bottleneck.\n\n## Reverse-engineering internet videos into interactions\n\nThe team reverse-engineers existing internet videos: they deduce logical user goals from the source footage, then segment the videos into multi-turn clips that simulate continuous interaction. To handle diverging interaction paths — the same goal can be achieved through many methods — models receive predefined priors derived from the source video and must guide users along the exact same routes.\n\nThe dataset contains 685 open-ended QA pairs across 7 major task types and 16 fine-grained tasks, spanning sports, cooking, lectures, DIY and talk shows. The four-phase annotation pipeline demands roughly 4 expert hours per sample — more than 1000 expert person-hours in total by the paper's account. Scoring uses an LLM-as-a-Judge pipeline on a 0-5 scale, normalized to 0-100.\n\nOne design detail stands out: user questions are never typed as text. They are embedded directly into the video — mostly as TTS audio, sometimes as subtitles, handwriting, or picture-in-picture gestures. The model sees only the video itself. Multi-turn simulation also preserves temporal causality: each turn's clip starts where the previous one ended, with the user prompt embedded at the very end.\n\n## The leaderboard: 66.4 at the top, no real winners\n\nEleven models were evaluated. Gemini-3-Pro leads with 66.4\u002F100, Gemini-2.5-Pro follows at 64.6, Doubao-Seed-2.0-lite takes third at 57.3, MiMo-V2-Omni scores 53.8 and Qwen3.5-Omni-Plus 51.6. The best open-source entrant, Qwen3-Omni-Instruct (30B-A3B), reaches only 51.2, while VITA-1.5 sits at the bottom with 24.6.\n\nThe team also filmed 3 real-world cases averaging 15 interaction turns (meeting simulation, blind-assisted navigation, and multi-person handicraft tracking). Here Gemini-2.5-Pro, second on the overall board, collapses to 44.8 — below Qwen3-Omni-Instruct at 53.8 — while Gemini-3-Pro holds first at 68.0. Offline leaderboard scores and genuine long-horizon interaction clearly diverge.\n\n## Three bottlenecks for the next generation\n\nThe paper identifies three recurring failure modes:\n\n- **Visual prompts**: models struggle to recognize hand gestures as user instructions; gesture-based prompt following broadly fails;\n- **Context exhaustion**: video and audio devour tokens so fast that, without long-term memory mechanisms, models exhaust the context window within minutes and then forget the user's goal;\n- **Delayed response**: models cannot tell when staying silent is the right move, frequently emitting loosely related video captions instead of waiting for the target event.\n\nBy the team's rubric, the 40-60 band means models \"generally understand what to do but struggle to provide accurate and comprehensive answers\" — which is where most of the leaderboard sits.\n\n## So what\n\nOmniAssistBench draws a clear line between video understanding and video collaboration. The former is passive clip-watching QA; the latter demands active alignment of visual state, user goals and prior knowledge, plus a sense of timing. The Real Cases divergence is the warning worth heeding: the gap between offline benchmark scores and real long-horizon interaction is wider than aggregate numbers suggest. If your product bets on an AI video assistant, running it on this bench before shipping beats watching launch-day demos.","omniassistbench-omni-llm-video-assistant","2026-08-21T17:59:52Z","2026-08-24T21:07:22.721546Z","2026-08-24T21:07:22.721561Z",true,"agent",33,{"items":39},[40,45,50,55,60,65],{"id":41,"title":42,"news_slug":43,"published_at":44},"7258978b-dfcd-4cb4-91c4-3b8569cd5deb","Qwen-Audio-3.0-TTS双版本发布:Plus登顶Artificial Analysis,Flash压到300ms首包延时","qwen-audio-3-tts","2026-07-20T10:00:00+00:00",{"id":46,"title":47,"news_slug":48,"published_at":49},"4c7f5330-3aff-458a-9ef5-f04cc5585703","微信视觉团队开源 WeMM 嵌入模型:2B 反超 8B 前基线,9B 达 MMEB-v2 80.6","wemm-embedding-wechat-multimodal","2026-08-26T21:07:30+00:00",{"id":51,"title":52,"news_slug":53,"published_at":54},"ea444bd9-4683-486b-b606-c222d98f1ba7","标注即 rollout:南开 OraRL 把视频多模态 RL 训练成本砍半,9B 空间智能超 GPT-5","orarl-annotations-as-rollouts-video-rl","2026-08-26T17:10:00+00:00",{"id":56,"title":57,"news_slug":58,"published_at":59},"ff0bc92a-295a-4707-be8d-76115fe9eeee","PerceptionBench 出炉:16 个前沿多模态模型,视觉感知无一及格","moonshot-perceptionbench-atomic-perception","2026-08-26T13:15:00+00:00",{"id":61,"title":62,"news_slug":63,"published_at":64},"12a5f49d-8c83-4c40-82af-1c0b7f1c8b3e","DeepSeek 给 V4-Flash 装上眼睛:Vision-Exp 实验模型两项基准反超 Opus 4.8","deepseek-v4-flash-vision-exp-multimodal","2026-08-21T23:05:00+00:00",{"id":66,"title":67,"news_slug":68,"published_at":69},"b95b93e8-294a-4c5b-b53d-ce6ea07c1519","SemComp-Bench 登顶 Hugging Face 日榜:视频生成开始考「任务做没做成」","semcomp-bench-video-task-completion","2026-08-20T13:30:00+00:00"]