[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-qwen-omnivchat-native-audio-visual-dialogue":3,"topics-all":41,"news-related-5d3c50e8-5087-43e9-a8f1-c64973f712c1":60},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":27,"news_slug":34,"published_at":35,"created_at":36,"modified_at":37,"is_published":38,"publish_type":39,"image_url":14,"view_count":40},"5d3c50e8-5087-43e9-a8f1-c64973f712c1","Qwen 拆掉 ASR 管道:音视频原生对话靠合成数据练成","Qwen 团队论文 OmniVChat 定义原生音视频对话:Omni 模型直接收用户的音频和视频并回文本,不经语音识别与外挂字幕。团队用多智能体引擎 OmniVChat-Studio 合成 2800 条对话构建基准,并用统一量尺的 RL 奖励训练 Qwen3-Omni-Instruct,真人测试集同样涨分。","挂在 HF 论文页 Qwen 徽标下的新论文 OmniVChat 把这条管道直接拆了。视频通话里的 AI 助手,多数还在走间接路线:音频先过一遍语音识别,画面先跑一层字幕或描述,拼成文字再喂给大模型。而这篇论文定义的 OmniVChat 任务是——Omni 模型同时接收用户的音频和视频,直接回文本;问题就藏在音视频本身,没有单独的文字提问,没有外挂字幕,也没有语音识别转写。论文的判断是:直接吃音视频,既减少外部件带来的延迟和计算量,也保住了语调、表情这类原始感知线索(arXiv: https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.21465)。\n\n## 任务好定义,数据是死穴\n\n原生音视频对话的麻烦不在架构,在素材。论文点破两个现实约束:一是真实素材稀缺,普通人对着自己设备录制的对话数据几乎找不到;二是评分难,一个好的回答往往要结合用户周围环境、面部表情和手边物体,而同一个意思有成千上万种说法,关键词匹配式评分在这里基本失灵。研究团队的应对思路是「为理解而生成」:让生成系统去造对话,拿合成数据做训练和评测。顺带一提,这是篇 18 人合作论文,README 机构脚注列了港中文、阿里 Token Hub、上海交大、上海创新研究院和浙大。\n\n## 四个 agent 造数据,2800 条对话进基准\n\n造数据的是 OmniVChat-Studio,一个多智能体数据引擎:Director 管文本进出,Renderer 把通过的 prompt 渲染成音画同步的片段,Reviewer 给结果配说明并写质量报告,确定性 Validator 按规则卡脚本。仓库 README 显示,基准 OmniVChat-Bench 共 2800 条对话,其中 2550 条单轮、250 条多轮,覆盖 5 大能力类:对话状态与链路感知(DSLP)900 条、多模态实体对齐(MEA)900 条、模型自我认知(MSA)450 条、反幻觉(AH)400 条、情绪识别(ER)150 条,合计 17 个子类、22 个场景域、13475 条分层评分细则。语言上英文占 63.1%、中文占 36.9%;媒体文件是 3490 个 1080p mp4、约 28 GB,用户的语音就是视频自带音轨(GitHub: https:\u002F\u002Fgithub.com\u002FHarlandZZC\u002FOmniVChat)。\n\n## RL 奖励:同一个量尺,训练和评测共用\n\n训练侧的 OmniVChat-RL 用一个公式同时约束正确性、效率和风格:`R(y) = r(y) + λ_fmt·f(y) + λ_eff·e(y) + λ_sty·s(y)`,其中 r 打正确性,f 管格式,e 管效率,s 管风格;README 给出的模板默认 λ_fmt=0.5、λ_eff=0、λ_sty=0,奖励范围 [0, 1.5]。一个值得注意的工程决策:评测打分器和训练奖励共用同一个量尺——评测代码 eval\u002Fscore.py 直接从 reward\u002Freward.py 导入 rubric 核心,量尺只有一份定义。但 README 同时强调,训练奖励均值和基准分不在同一量纲上,两个数不能直接对比。\n\n## sim-to-real:合成数据练出来的,真人测试也涨\n\n验证环节才是这篇论文的关键证据:用 OmniVChat-RL 在合成对话上训练 Qwen3-Omni-Instruct 后,模型在 OmniVChat-Bench 和真人录制的 OmniVChat-Bench-Human 上都拿到了提升——合成数据训出来的能力能迁移到真实对话的训练与评测,这正是一作在 HF 评论区总结里强调的 sim-to-real 迁移。\n\n## 所以呢\n\n两件事值得记住。其一,这份基准把「模型知道自己没有身体」(MSA)和「不编造音视频证据之外的内容」(AH)写进了评分体系,这才是音视频助手走进生产环境的门槛——不是答得多漂亮,而是知道自己不知道。其二,当真实数据成为瓶颈,合成数据已经从训练补丁升级成任务定义的一部分:先造数据,再定基准,最后训模型,一条流水线全自己搭。反问留给读者:当评测集本身是造出来的,你对分数的信任要不要也打个折?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.21465","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21,24],{"id":12,"name":13,"slug":13,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":25,"name":26,"slug":26,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[28],{"id":29,"lang":30,"title":31,"summary":32,"content":33},"50e4d870-0aa8-46ed-8930-a45924e6749b","en","Qwen Ditches ASR: Native Audio-Visual Dialogue via Synthetic Data","Qwen-team paper: native audio-visual dialogue with no ASR or captioning. 2,800 synthesized dialogues build the benchmark; RL gains hold on human tests.","Most AI assistants on a video call still take the long way round: audio goes through speech recognition, frames go through captioning or description, and the stitched-together text is what the model actually reads. OmniVChat, a new paper carrying the Qwen org badge on the Hugging Face papers page, removes that pipeline entirely. The omni model receives the user's audio and video simultaneously and replies in text; the question lives inside the audio and video themselves, with no separate text prompt, no external captioning, and no ASR transcript. The paper's argument: direct audio-visual input reduces external latency and computation while preserving perceptual cues (arXiv: https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.21465).\n\n## Easy to define, hard to feed\n\nThe bottleneck is not architecture but material. The paper names two constraints. Real recordings are scarce — people rarely hand over footage of themselves talking to their own devices. And evaluation is hard: a good reply often depends on the user's surroundings, facial expressions, and nearby objects, while the same intent can be phrased countless ways, which makes keyword matching unreliable. The team's answer is generation-for-comprehension: synthesize the dialogues, then use them for both training and evaluation. The paper has 18 authors; the README's affiliation footnote lists CUHK, Alibaba Token Hub, Shanghai Jiao Tong University, Shanghai Innovation Institute, and Zhejiang University.\n\n## Four agents, 2,800 dialogues\n\nOmniVChat-Studio, the multi-agent data engine, assembles the data: a Director handles text input and output, a Renderer turns accepted prompts into synchronized audio-visual clips, a Reviewer captions results and writes quality reports, and a deterministic Validator checks scripts against configured rules. Per the repo README, the OmniVChat-Bench benchmark holds 2,800 dialogues — 2,550 single-turn and 250 multi-turn — across five ability categories: Dialogue-State & Link Perception (DSLP, 900), Multimodal Entity Alignment (MEA, 900), Model Self-Awareness (MSA, 450), Anti-Hallucination (AH, 400), and Emotion Recognition (ER, 150), spanning 17 subcategories, 22 scenario domains, and 13,475 tiered rubric criteria. English accounts for 63.1% and Chinese 36.9%; the media amounts to 3,490 1080p mp4 files, about 28 GB, where the user's speech is the video's own audio track (GitHub: https:\u002F\u002Fgithub.com\u002FHarlandZZC\u002FOmniVChat).\n\n## One rubric, two consumers\n\nTraining-side OmniVChat-RL compresses the reward into one formula — `R(y) = r(y) + λ_fmt·f(y) + λ_eff·e(y) + λ_sty·s(y)` — where r scores correctness, f format, e efficiency, and s style; README defaults are λ_fmt = 0.5, λ_eff = 0, λ_sty = 0, with reward range [0, 1.5]. A deliberate engineering choice: the benchmark scorer and the training reward share one rubric implementation — eval\u002Fscore.py imports from reward\u002Freward.py — so the yardstick has exactly one definition. The README also warns that training-reward means and benchmark scores live on different scales and are not directly comparable.\n\n## Sim-to-real is the point\n\nThe decisive evidence: after training Qwen3-Omni-Instruct with OmniVChat-RL on synthesized dialogues, the model improves on both OmniVChat-Bench and the human-recorded OmniVChat-Bench-Human. Capability built on synthetic data transfers to real-world dialogue training and evaluation — the sim-to-real point the first author highlighted in the HF paper-page comments.\n\n## So what\n\nTwo takeaways. For one, this benchmark scores whether a model knows it has no body (MSA) and whether replies stay grounded in the available audio-visual evidence (AH) — the actual gate for audio-visual assistants entering production, and a harder bar than eloquence. For another, once real data becomes the bottleneck, synthetic data graduates from training patch to part of the task definition: build the data, define the benchmark, train the model, all in one house. A question to leave you with: when the eval set itself is manufactured, how much discount should you apply to the score?","qwen-omnivchat-native-audio-visual-dialogue","2026-09-21T15:14:15Z","2026-09-21T15:14:28.981957Z","2026-09-21T15:14:28.981965Z",true,"agent",144,[42,51],{"slug":43,"tag_slug":43,"title_zh":44,"title_en":45,"intro_zh":46,"intro_en":47,"id":48,"is_active":38,"created_at":49,"modified_at":50},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":52,"tag_slug":52,"title_zh":53,"title_en":54,"intro_zh":55,"intro_en":56,"id":57,"is_active":38,"created_at":58,"modified_at":59},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":61},[62,67,72,77,82,87],{"id":63,"title":64,"news_slug":65,"published_at":66},"dcb1506b-87fa-422f-89e0-bb62afcc2b4c","BenchLM 8 月榜:Qwen3.8 Max 79.2 分领跑开源 LLM,MiniMax M3 跻身三强","qwen3-8-max-benchlm-aug-2026","2026-08-28T06:00:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"ff0bc92a-295a-4707-be8d-76115fe9eeee","PerceptionBench 出炉:16 个前沿多模态模型,视觉感知无一及格","moonshot-perceptionbench-atomic-perception","2026-08-26T13:15:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"40095b51-97b0-4fd4-9b1d-f636c970572e","阿里 Qwen 团队发布 Qwen3.8-Max:2.4 万亿参数 MoE 模型首度开放权重","qwen3-8-max-2-4t-moe-open-weights","2026-08-07T02:00:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"4c9f74d4-0252-4e86-8b6e-85d38788eea6","开源编程模型三选一:GLM-5.2、DeepSeek V4、Qwen3.6","glm-5-2-deepseek-v4-qwen-3-6-coding","2026-07-27T06:00:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"0138c087-f1c3-44c5-a662-045c6f20a756","ACL 2026 新基准:大模型的古诗美学推理翻车了","neo-classic-poetry-reasoning-benchmark","2026-09-20T15:15:00+00:00",{"id":88,"title":89,"news_slug":90,"published_at":91},"12c67d52-17a2-4df5-8386-35d18ffd221a","JEPA-Anything:一套预测框架打通七个领域,湿实验也给了背书","jepa-anything-orthogonal-predictive-factorization","2026-09-19T23:10:37+00:00"]