[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-dqwen3-5-hybrid-attention-diffusion-language-models":3,"topics-all":38,"news-related-4455b8ee-eab9-463b-9934-f1df4b1b4fb3":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"4455b8ee-eab9-463b-9934-f1df4b1b4fb3","扩散语言模型的适配断点被接上:dQwen3.5 只花一半 token","UT Austin 团队把 Qwen3.5 四个尺寸改造成混合注意力扩散语言模型:只双向化注意力层,GDN 循环层保持因果。达到同样训练损失只需约一半 token,9B 版 50B token 在 7 项基准中 4 项超过 580B token 的 Dream-7B,全家族开源。","扩散语言模型（DLM）的经济账一直很难看：不从零训练、直接改造一个现成的自回归（AR）模型，是公认的省钱路线。但 UT Austin 的一个六人团队注意到一个被整个领域绕开的错位——几乎所有 AR-to-DLM 适配研究都从全注意力 Transformer 出发，而现实中新一代开源模型早就不是纯全注意力了。\n\n## 混合架构才是现实，但它对 DLM 适配不友好\n\nQwen3.5 的序列处理层里约四分之三是 Gated DeltaNet（GDN）这样的循环层，只有四分之一是注意力层。循环层天生是因果的——状态从左往右流，想让它「双向看」整个序列，结构上就说不通。这正是过去适配研究默认用全注意力 backbone 的原因：注意力层双向化是改个掩码的事，RNN 不是。\n\n团队的解法反直觉地简单：不双向化 RNN。dQwen3.5 的改造只把注意力层改成双向，GDN 层原样保留因果结构，再配合 token shifting——让位置 k 的隐状态仍然预测位置 k+1 的 token，保持与 AR 预训练一致的读出对齐。\n\n他们把 Qwen3.5 的 0.8B\u002F2B\u002F4B\u002F9B 四个尺寸按同一套 recipe 改造，称为 dQwen3.5 家族；另把 Qwen3-1.7B（全注意力）改造成 trunk 参数量对齐的对照组。训练分 50B 和 100B token 两档，batch 512 序列 × 4096 token，bf16 精度。\n\n## 数字：一半 token，以及 50B 打 580B\n\n两个层面的结果值得记：\n\n- **效率**：对照 trunk 对齐的全注意力对照组，混合 backbone 的 dQwen3.5-2B 达到同样训练损失只需要约一半 token。\n- **效果**：dQwen3.5-9B 只用 50B token 改造，在与 Dream-7B（580B token）、Dream-Coder-7B（322B）、LLaDA-8B（2.3T、从零训练）的对比里，7 项基准拿了 4 项最高分；2B 版 50B token 在 7 项里 6 项超过用 200B token 训练的 CoDA。\n\n## 诚实的另一面\n\n在知识与代码的天平上，改造是有代价的：0.8B 上原版 MMLU 50.31，50B token 适配版 32.62——知识类掉得明显，代码类反而涨（HumanEval 22.56 → 28.05）。「训得越久越好」也不成立：50B 加到 100B，0.8B 上 7 项基准 5 项提升，9B 上只有 1 项。RNN 的因果结构也没有杀死 DLM 的灵魂：any-order 解码与全注意力 DLM 一致（局部 AR-ness 0.58–0.66，落在所有被测 DLM 同一区间），并行解码高加速比下 HumanEval 保持度还优于同规模对照。\n\n## 开源与工程细节\n\ndQwen3.5 全家族（0.75B\u002F1.88B\u002F4.21B\u002F8.95B 四个 hybrid，加 1.72B 全注意力对照）都在 Hugging Face 的 UT-IFML 组织下开源，代码在 GitHub（AntonXue\u002FdQwen）。使用门槛不低：需要 transformers≥5.13、flash-linear-attention 0.5.1；推理是整 canvas 一次解码、置信度阈值 tau=0.9 时提交位置，或用 block_length=32 做分块左到右解码。写稿时 9B 版月下载 543——当前状态数字，仅供参考。\n\n## 所以呢\n\n这篇论文的真正贡献不是刷分，而是把 DLM 适配研究从「全注意力惯性」拉回到「现实中真实流行的 hybrid backbone」。当 Jamba、MiniMax、Qwen 这些主流模型都在往 attention+RNN 混合架构迁移时，DLM 社区如果只会在全注意力模型上做适配，路线本身就是脱节的。现在这条断头路接上了，而且带着四个尺寸的开源权重和一套可复现的 recipe——对任何想低成本持有 DLM 的团队，这是目前最现实的起点。\n\n（引用：[arXiv:2609.20751](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.20751) · [UT-IFML\u002FdQwen3.5-9B-Base](https:\u002F\u002Fhuggingface.co\u002FUT-IFML\u002FdQwen3.5-9B-Base)）","https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.20751","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"7b67033c-19e6-4052-a626-e681bba64c7a","diffusion",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b9bd9039-fcdb-41a8-b85b-fc1587def2b9","open-source",{"id":22,"name":23,"slug":23,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"5dd7a851-2f4c-4832-aae0-e923034949d4","en","dQwen3.5 Adapts Hybrid Qwen Backbones into Diffusion LMs","UT Austin adapts Qwen3.5 into diffusion LMs; only attention goes bidirectional. Half the tokens to match loss; 9B at 50B tokens beats Dream-7B's 580B.","The economics of diffusion language models (DLMs) have always looked bad: adapting a pretrained autoregressive (AR) model instead of training from scratch is the accepted cost-saving route. But a six-person team at UT Austin noticed a mismatch the entire field had been steering around — nearly all AR-to-DLM adaptation research starts from a full-attention transformer, while real-world frontier open models abandoned pure full attention long ago.\n\n## Hybrid backbones are the reality, and they resist DLM adaptation\n\nAbout three-quarters of Qwen3.5's sequence-processing layers are recurrent Gated DeltaNet (GDN) layers; only a quarter are attention. Recurrent layers are structurally causal — state flows left to right, and \"looking both ways\" over the whole sequence is architecturally off the table. That is exactly why past adaptation work defaulted to full-attention backbones: bidirectionalizing attention is a masking change; an RNN is not.\n\nThe team's fix is counterintuitively simple: don't bidirectionalize the RNN. dQwen3.5 only makes its attention layers bidirectional, leaves the GDN layers causal as-is, and applies token shifting — the hidden state at position k still predicts the token at position k+1, preserving the readout alignment from AR pretraining.\n\nThey adapted Qwen3.5 at 0.8B\u002F2B\u002F4B\u002F9B scales under one shared recipe, calling the result the dQwen3.5 family; a full-attention Qwen3-1.7B was adapted as a trunk-matched control. Training ran at 50B and 100B token budgets, batch 512 sequences × 4096 tokens, bf16 precision.\n\n## The numbers: half the tokens, and 50B beats 580B\n\nTwo layers of results are worth remembering:\n\n- **Efficiency**: against the trunk-matched full-attention control, the hybrid dQwen3.5-2B reaches a given training loss in about half the tokens.\n- **Effectiveness**: dQwen3.5-9B after only 50B adaptation tokens takes the highest score on 4 of 7 benchmarks against Dream-7B (580B tokens), Dream-Coder-7B (322B), and LLaDA-8B (2.3T, trained from scratch); at 2B, 50B tokens beat CoDA (200B tokens) on 6 of 7 benchmarks.\n\n## The honest flip side\n\nOn the knowledge-vs-code balance, adaptation has a cost: on the 0.8B model, original Qwen3.5 scores 50.31 MMLU while the 50B-token adapted version manages 32.62 — knowledge-heavy capability drops sharply, while code benchmarks actually rise (HumanEval 22.56 → 28.05). \"Longer is better\" also fails: going from 50B to 100B improves 5 of 7 benchmarks at 0.8B but only 1 of 7 at 9B. And the causal RNN structure does not kill the DLM's soul: dQwen3.5 matches full-attention DLMs in any-order decoding behavior (local AR-ness of 0.58–0.66, inside the same range as every other DLM evaluated), and under parallel decoding it retains more HumanEval performance at higher speedup ratios than comparable-trunk peers.\n\n## Open weights and engineering details\n\nThe full dQwen3.5 family (0.75B\u002F1.88B\u002F4.21B\u002F8.95B hybrids plus a 1.72B full-attention control) is open on Hugging Face under the UT-IFML org, with code on GitHub (AntonXue\u002FdQwen). The bar is not low: transformers>=5.13 and flash-linear-attention 0.5.1 required; inference decodes the whole canvas at once, committing positions above confidence threshold tau=0.9, or uses block_length=32 for block-wise left-to-right decoding. At writing time the 9B model shows 543 downloads last month — a current-state number, for reference only.\n\n## So what\n\nThe real contribution here is not benchmark points but dragging DLM adaptation research from the \"full-attention habit\" back to the hybrid backbones that actually dominate shipping models. As Jamba, MiniMax, and Qwen all migrate toward attention+RNN hybrids, a DLM community that only adapts full-attention models is building on a route that no longer exists. That dead end is now connected — with open weights at four sizes and a reproducible recipe, this is the most practical starting point for any team that wants to own a DLM cheaply.\n\n(References: [arXiv:2609.20751](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.20751) · [UT-IFML\u002FdQwen3.5-9B-Base](https:\u002F\u002Fhuggingface.co\u002FUT-IFML\u002FdQwen3.5-9B-Base))","dqwen3-5-hybrid-attention-diffusion-language-models","2026-09-18T19:20:00Z","2026-09-18T19:12:51.979752Z","2026-09-18T19:12:51.979770Z",true,"agent",102,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"d056f67b-7e0d-4e44-8d39-e31ea50deeae","Bonsai 2 27B 三元压缩:Qwen3.8 压到 5.9 GB,benchmark 留存 98.2%","bonsai-2-27b-ternary-qwen3-8-compression","2026-09-17T15:47:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"d941056b-c2e7-42e5-965a-a982c20b1169","Qwen3.8-Flash-Next 架构细节:Gated Residual 多分支残差 + QSA micro-block 稀疏注意力","qwen3-8-flash-next-cost-efficiency-architecture","2026-09-02T02:00:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"637f84e0-e6dc-490a-bba1-879f6527bdd5","Qwen3.8-Max 2.4T 开源:Gated DeltaNet 把长上下文成本砍到 1\u002F8","qwen3-8-max-2-4t-open-weights-gated-deltanet","2026-08-30T03:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"99cc91d0-a9b1-46c8-b633-2b79a0dcdcb2","HF 夏季报告:开源 LLM 的点赞与下载,其实是两个市场","hf-summer-2026-likes-vs-downloads","2026-08-28T08:00:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"dcb1506b-87fa-422f-89e0-bb62afcc2b4c","BenchLM 8 月榜:Qwen3.8 Max 79.2 分领跑开源 LLM,MiniMax M3 跻身三强","qwen3-8-max-benchlm-aug-2026","2026-08-28T06:00:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"0d8fdf45-4585-47c0-9e78-3652e318b156","Apple Intelligence 中国版落地:通义千问接管语言 AI,百度负责视觉搜索","apple-intelligence-china-qwen-baidu-2026","2026-08-25T12:00:00+00:00"]