[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-full-duplex-speech-collapse-fd-lhb-kv-cache":3,"topics-all":38,"news-related-e4ac848d-d24a-465f-bece-37e090da86de":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"e4ac848d-d24a-465f-bece-37e090da86de","全双工语音模型长对话必崩:KAIST 揪出初始化锚点被挤走","KAIST 与庆熙大学团队发布 FD-LHB 基准,用 40 段真实人声回放和 40 场 LLM 对话测五个全双工语音模型,结果无一能在长对话中保持稳定。论文把崩溃归因于滑窗缓存挤走首个 KV 条目与模型复读自身输出,并给出无需重训的 IAP 与 DRP 两个补丁。","想象你跟一个全双工语音助手聊了十分钟：它一边听你说话一边出声回复，前五分钟自然流畅，然后开始重复自己的词，接着突然安静，再然后吐出一串没人能听懂的音节。这不是个别模型的 bug，而是一整类模型在长对话里的共同命运。\n\nKAIST 集成视觉与语言实验室与庆熙大学(Kyung Hee University)的研究团队在 arXiv 公开了 Full-Duplex Long-Horizon Benchmark（FD-LHB），专门用来把这个问题逼出来（[arXiv:2610.32349](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32349)，[HuggingFace 论文页](https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2610.32349)）。基准分两条轨道：一条是 FD-LHB-CANDOR，从 CANDOR 真人长对话语料里去掉开头三轮、按时长排序取最长 40 段，用同一段录音回放给每个模型，最多跑到 60 分钟；另一条是 FD-LHB-LLM，让 Gemini-2.5-Flash 担任对话搭档，本地 mms-tts-eng 把它的回复合成为 16 kHz 语音，每个模型跑 40 场、每场 20 分钟，话题覆盖兴趣、旅行、家庭、工作等十类日常场景。回放轨保证可控，交互轨保证真实，两条互相补位。\n\n## 五款模型，两种死法\n\n被测的五款模型是 Moshi、PersonaPlex、Raon-SpeechChat、BayLing-Duplex 和 Lychee-FD。前两款用滑窗缓存，3000 帧窗口在 12.5 Hz 下大约四分钟就填满；后三款是增长式上下文，跑满上下文分别需要约 5、17.5 和 10.9 分钟。\n\n论文口径的数据是：没有任何一款模型撑住了几十分钟的对话。CANDOR 回放轨上，Moshi 和 PersonaPlex 在 60 分钟内的重复、长时静默、胡言乱语出现比例分别是 65%\u002F100%\u002F100% 和 90%\u002F85%\u002F100%；BayLing-Duplex 在 17.5 分钟内三项分别是 55%、100%、90%；Lychee-FD 在 10.9 分钟内出现重复和胡言的比例是 37.5% 和 30%；Raon-SpeechChat 则在约五分钟时上下文耗尽。在有来有回的 FD-LHB-LLM 上，基线模型最常见的死法是胡言乱语，命中 97.5%–100% 的场次，长时静默命中 70%–87.5%。\n\n值得注意的是\"上下文耗尽\"被单独算作运行结束，不计入塌缩。也就是说滑窗模型属于\"内存永远够用，但对话照样崩\"。\n\n## 病根一：滑窗把首个 KV 条目挤走了\n\n论文给了一个相当反直觉的诊断。训练时样本被切成块，每块都从空缓存开始，于是每一块开头都有一个\"初始化锚点\"——首个在无任何历史缓存下算出来的 KV 条目。但推理时缓存从不清空，窗口一满就淘汰最旧的条目，锚点迟早被挤掉，模型就进入了训练时从未见过的状态。\n\n证据链很硬：改变注意力窗口长度 W，静默出现时间按 (1.63±0.49)W 增长，胡言乱语按 (2.70±0.20)W 增长；而在纯静音输入下胡言仍按 (2.54±0.19)W 出现，说明这跟对话内容无关、只跟缓存周转有关。更直接的实验是只保留一个早期条目——第 1、2、3、4、5、9、17 个，或者 L\u002F4、L\u002F2、3L\u002F4 这些位置——结果只有首个条目能在所有轮次里挡住胡言，其余每一个选择都至少在一次运行中失败；保留前 P 个条目时，P=0 全部崩溃，P≥1 的曲线完全重合。\n\n## 病根二：模型在复读自己\n\n重复是另一套机制。论文观察到重复期最高频的词是 of、then、you 这类短高频词，以及 yeah、okay 这类应答语；每轮运行里占主导的那个词，在重复真正开始之前就已经在爬升，之后继续涨。作者还往上下文里插入一个原本不存在的词 purple，统计接下来 20 秒它被重复的次数——插入越多，复发越多。\n\n## 不用重训的两个补丁\n\n顺着诊断，论文给出两个纯推理期的干预：Initialization-Anchor Pinning 把锚点钉住，只更新它自己那一个条目的位置索引；Duplex Repetition Penalty 对最近 512 个词元内出现过的词做惩罚，同时把 PAD\u002FEPAD 排除在外，并在长时间静默时逐渐压低 PAD 的 logit，让模型还能继续开口。\n\n效果是：回放轨上 Moshi 三项塌缩降到 0%、7.5%、0%，PersonaPlex 降到 2.5%、47.5%、0%；交互轨上每种失败都不超过 10% 的场次。代价是论文实测的实时因子——基线 0.5957，补丁方案 0.7525，而 StreamingLLM 式方案 1.0323 已经跟不上实时对话。消融实验显示两者互补：IAP 单独就把 Moshi 的胡言清零，DRP 单独不管用，而静默这一项能从 0% 提到 67.5%、再加 DRP 到 92.5%。\n\n## 这个结论的边界\n\n论文自己写得很清楚：诊断只覆盖两款滑窗模型，增长式上下文模型同样会塌，但原因不明，IAP 也不适用；FD-LHB-LLM 每个模型只跑 40 场，胡言判定依赖 Gemini-2.5-Flash 评委，8 人盲听验证显示评委不会误判正常语音、但会漏掉一部分真胡言，所以真实比例可能被低估；补丁也未能消灭全部问题，PersonaPlex 上仍有 19\u002F40 场出现两分钟静默；15 人的听感实验显示对话早期听众其实更偏好不加 DRP 的版本（0.31–0.43），中后期才反过来。\n\n所以呢：如果你的产品里有全双工语音客服，最该做的不是换一个更大的模型，而是拿一段二十分钟的真实录音回放一遍，看看它从第几分钟开始重复、开始沉默、开始胡说。塌缩是可以在上线前测出来的，而且大多不需要重训就能修掉大半。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32349","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"40269b40-7942-4650-9672-ed2e6524d37a","ai-technology",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"120fa59a-ff6f-4537-9bf5-f818df636a0e","benchmark",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"499f4b56-819d-49a3-9609-33e775143b86","multimodal",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"630457dd-4fb6-4bfc-b01d-960fd2ff7a41","en","Full-Duplex Speech Models Collapse in Long Conversations","FD-LHB runs five full-duplex speech models through 40 replays and 40 live LLM chats: all collapse, from KV-cache eviction and self-reinforcement.","Imagine talking to a full-duplex speech assistant for ten minutes: it listens and speaks at the same time, and the first five minutes flow naturally. Then it starts repeating its own words, then it goes silent, then it emits a stream of syllables nobody can make sense of. That is not one model’s bug. It is the shared fate of an entire class of models in long conversations.\n\nA team from KAIST’s Integrated Vision and Language Lab and Kyung Hee University released the Full-Duplex Long-Horizon Benchmark (FD-LHB) on arXiv to force this problem into the open ([arXiv:2610.32349](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.32349), [HuggingFace paper page](https:\u002F\u002Fhuggingface.co\u002Fpapers\u002F2610.32349)). The benchmark has two complementary tracks. FD-LHB-CANDOR takes the CANDOR corpus of long, natural human conversations, removes the first three turns to exclude greetings and initial silence, ranks what remains by duration, and keeps the longest 40, replaying the same fixed audio to every model for up to 60 minutes. FD-LHB-LLM is interactive: Gemini-2.5-Flash plays the speaking partner, a local facebook\u002Fmms-tts-eng VITS model turns its text into 16 kHz speech, and every dialogue model runs 40 sessions of up to 20 minutes across ten everyday topics such as hobbies, travel, family, and work. The replay track guarantees controlled comparison; the interactive track guarantees realism. Each compensates for the other’s weakness.\n\n## Five models, two ways to die\n\nThe five models evaluated are Moshi, PersonaPlex, Raon-SpeechChat, BayLing-Duplex, and Lychee-FD. The first two use sliding-window caches: a 3,000-frame window at 12.5 Hz fills after roughly four minutes. The other three grow their context until it is full, after about 5, 17.5, and 10.9 minutes respectively.\n\nBy the paper’s own numbers, none of them held a conversation together for tens of minutes. On FD-LHB-CANDOR, Moshi and PersonaPlex showed repetition, prolonged silence, and gibberish in 65%\u002F100%\u002F100% and 90%\u002F85%\u002F100% of runs over 60 minutes. BayLing-Duplex hit 55%, 100%, and 90% within 17.5 minutes; Lychee-FD showed repetition and gibberish in 37.5% and 30% of runs within 10.9 minutes; Raon-SpeechChat exhausted its context after about five minutes. On FD-LHB-LLM, where the partner actually responds, the baselines most often failed through gibberish, hitting 97.5%–100% of sessions, and prolonged silence, hitting 70%–87.5%.\n\nOne detail matters here: reaching context capacity is counted as the end of a run, not as a collapse event. The sliding-window models therefore fall into the category of “memory is never the bottleneck, and the conversation still falls apart.”\n\n## Cause one: the sliding window evicts the first KV entry\n\nThe diagnosis is counter-intuitive. During training, samples are split into chunks, and each chunk starts from an empty cache, so every chunk begins with an initialization anchor: the first KV entry, computed without any earlier cached state. At inference the cache is never reset. Once it is full, the oldest entries are evicted, the anchor goes with them, and the model enters a state it never saw in training.\n\nThe evidence chain is tight. Varying the attention-window length W, silence onset grows at (1.63±0.49)W and gibberish onset at (2.70±0.20)W. With silence-only input, gibberish still appears and still grows at (2.54±0.19)W, which ties the failure to cache turnover rather than to conversational content. The sharper experiment keeps exactly one early entry, varying only which one: the 1st, 2nd, 3rd, 4th, 5th, 9th, or 17th entry, or the entry at L\u002F4, L\u002F2, or 3L\u002F4. Only the first entry prevents gibberish in all runs; every other choice fails in at least one run. Keeping the first P consecutive entries tells the same story: with P=0 every run reaches gibberish, with any P≥1 no run does, and the curves overlap exactly.\n\n## Cause two: the model rehearses its own output\n\nRepetition runs on a separate mechanism. The most repeated words during repetition periods are short, frequent words such as of, then, and you, plus backchannels like yeah and okay. For each run, the word that dominates during repetition starts rising before the onset and keeps climbing afterwards. The authors also inserted purple, a word absent from the prior context, and counted how often the model repeated it over the next 20 seconds: recurrence increased with more insertions, most strongly in PersonaPlex.\n\n## Two fixes that need no retraining\n\nGuided by that diagnosis, the paper proposes two inference-time interventions. Initialization-Anchor Pinning keeps the anchor in the cache and updates only that single entry’s position index. Duplex Repetition Penalty penalizes any word that appeared in the model’s last 512 lexical tokens, excluding the PAD and EPAD tokens used for listening and turn-taking, and when the model has been silent for a long time while the user is quiet, it gradually lowers the PAD logit so the model can resume speaking.\n\nThe results: on the replay track, Moshi’s three collapse rates fall to 0%, 7.5%, and 0%, and PersonaPlex’s to 2.5%, 47.5%, and 0%. On the interactive track, every failure mode occurs in no more than 10% of sessions. The cost is the measured real-time factor: the baseline sits at 0.5957, the patched version at 0.7525, while a StreamingLLM-style baseline reaches 1.0323 and can no longer keep up with real-time conversation. The ablation shows the two components are complementary. IAP alone keeps every Moshi run free of gibberish while DRP alone does not help; silence-free runs go from 0% to 67.5% with IAP and to 92.5% once DRP is added.\n\n## Where the claim stops\n\nThe paper is explicit about its limits. The diagnosis covers only the two sliding-window models; growing-context models collapse too, but their cause is unidentified and IAP does not apply to them because they never evict entries. Each model runs only 40 sessions on FD-LHB-LLM, and gibberish is judged by a Gemini-2.5-Flash judge whose blind validation with eight raters showed no false positives but some misses, so the reported gibberish rates may underestimate the truth. The fixes do not eliminate everything either: PersonaPlex still produces a two-minute silence in 19 of 40 replays. And a blinded listening study with fifteen participants shows that early in a conversation, listeners actually prefer the version without DRP (0.31–0.43), with the preference flipping only later on.\n\nSo what does it mean for you? If your product has a full-duplex voice agent, the highest-value next step is not swapping in a bigger model. It is replaying twenty minutes of real recorded audio and watching for the minute when the model starts repeating, then going silent, then drifting into gibberish. This collapse is measurable before you ship, and most of it can be fixed without retraining.","full-duplex-speech-collapse-fd-lhb-kv-cache","2026-10-27T03:20:00Z","2026-10-27T03:23:35.524618Z","2026-10-27T03:23:35.524633Z",true,"agent",206,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"58aafc2c-b297-4f8c-a817-cad6ed7f1eda","跨模态检索头:18个模型实测,砍掉后音频检索归零","mllm-retrieval-heads-cross-modal-optimization","2026-11-01T17:09:14+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"a2ec5815-edbe-4467-a6ed-e3251b09023a","多轮越狱防御新解:把浅层对齐缺陷改成防御支点","shallowdefend-multi-turn-jailbreak-defense","2026-11-01T11:20:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"2f9ad534-b737-490e-b97e-bf01f2b6d26b","交错生成新基准:闭源旗舰文本赢,图像一致性输给开源","intreal-interleaved-generation-benchmark","2026-11-01T05:27:08+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"1b51020b-9a0c-46f8-98ec-e4e0c8c051a6","基准分其实是难度的加权平均:25 个主流基准的体检报告","benchmark-difficulty-measure-coverage-gaps","2026-11-01T01:16:42+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"fd0e593c-bb2c-44f9-8913-16f8dcef6999","苹果8B智能体:真实iOS上追平Gemini-3.1-Pro","apple-online-rl-ui-agent-live-ios","2026-10-31T11:10:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"8a550513-ef18-4d19-b609-d5fee5a853af","LeCun证明柏拉图表示定理:109M JEPA对齐胜72B","lecun-platonic-representation-theorem-jepa","2026-10-30T23:11:14+00:00"]