[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-hybrid-llm-training-free-sparse-decoding":3,"topics-all":38,"news-related-931283f8-6928-4128-bc6c-2d11ba240fca":57},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"931283f8-6928-4128-bc6c-2d11ba240fca","397B混合LLM免重训:长文本解码提速2.2倍","混合架构把多数层换成线性注意力,但剩下的全注意力层仍在长上下文解码时每步读完整KV。新论文提出免训练方案:滑窗+钉住前缀+query感知top-k,逐层用NSGA-II搜配置,装进vLLM。六个模型21个数据集精度保持0.5pp内,64k\u002F256k解码最高提速1.26\u002F2.23倍。","混合架构本来是给长上下文降成本的答案——把大部分全注意力层换成线性注意力,每步开销就不再随上下文线性膨胀。但有一笔账很多人没算过:剩下的那几个全注意力层,在 256k token 的语境里照样每步读完整 KV,反而成了长文本解码的新瓶颈。11 月 4 日挂上 arXiv 的一篇论文给了一个不动权重的解法:不重训、不改任何参数,只靠逐层搜索稀疏注意力配置,就让 Qwen3.5-122B-A10B 在 256k 上下文解码提速 2.23 倍,而且 21 个长上下文数据集精度不掉。\n\n## 剩下的全注意力层,才是长上下文的账单\n\n标准自注意力每生成一个 token,成本都比上一个更高:时间复杂度随序列长度平方增长,KV cache 随 token 数和并发请求数线性膨胀。混合模型的思路是把多数层换成廉价注意力,但留多少个全注意力层是预训练时焊死在权重里的:Qwen3.5-27B 是 64 层里留 16 层,gpt-oss-120b 是 36 层里留 18 层,Ornith-397B 是 60 层里留 15 层。这些层每步都要读全部上下文的 key,在长上下文任务里吃掉解码时间的大头,也决定了部署端能开多少并发。\n\n## 方法:滑窗 + 前缀 + query 感知选页,逐层搜配置\n\n论文的方案是一套组合式稀疏注意力,每个全注意力层只看三部分:近期 token 的滑动窗口、一段钉住的前缀、以及按 query 动态挑出的固定预算\"最相关 token\"(基于分页 KV 的 Quest 式选择)。\n\n真正的关键在\"逐层\"二字:窗口多大、预算多少,不是拍脑袋定的,而是用 NSGA-II 多目标搜索(精度 vs 估计成本)为每一层单独搜出来的,搜索精度目标用 BABILong-qa1(64k)和 LongBench Qasper 的前 100 个样本估计。这套搜索不便宜但可控:4B 到 120B 的模型在 A100 上花 8.6 到 34.2 GPU 时,122B 和 397B 换 4 张 GB200 的节点,也就 31.3 到 58.4 GPU 时。实现直接装进 vLLM,推理引擎不用换。\n\n两个诚实的限定:prefill 完全不动;KV cache 内存一 byte 都不省——省的是每个解码步要读的 key 数量,是时间,不是显存。\n\n## 结果与边界:白拿的 2 倍,和拿不到的地方\n\n评测覆盖六个模型:Qwen3.5-4B\u002F27B\u002F122B-A10B、Ornith-1.0-397B、Gemma-4-31B、gpt-oss-120b,跑 21 个长上下文数据集(BABILong qa1–qa5 的 64k 档加 LongBench 的 16 个英文子集),并且剔除了搜索阶段用过的样本,评测全部落在 held-out 数据上。\n\n结果是:每个模型都存在至少一套配置,在全部数据集上保住基线精度,平均差距不超过 0.5 个百分点。提速在 64k\u002F256k 上下文(batch 4)分别为:Qwen3.5-4B 1.26\u002F2.22 倍,Qwen3.5-122B-A10B 1.20\u002F2.23 倍,Ornith-1.0-397B 1.16\u002F2.06 倍——三个收益最好的模型,256k 档全部翻倍。\n\n边界同样写得很清楚:短上下文下,所有配置都比原模型更慢;稠密模型收益有限——Qwen3.5-27B 在 64k 只有 1.11 倍,Gemma-4-31B 在 128k 是 1.22 倍,因为 27B 模型 64k 解码一步 47.1ms 里 19.4ms 花在读权重上,这部分注意力优化救不了;gpt-oss-120b 也只有 1.16 倍(128k),因为搜出来的配置保留了它大部分全注意力层。\n\n## 为什么值得注意\n\n对比一下重训路线的成本:LoLCATs 给 Llama 3.1 405B 做注意力迁移花了 454 个 H100 GPU 时;Jet-Nemotron-2B 拿到 64k 12 倍提速,代价是 600B token 的训练量和 18,328 个 H100 GPU 时。本方法按 GPU 时计价的配置搜索,意味着每出一个新模型都可以重搜一遍配置,摊销逻辑完全不同。\n\n另一个有意思的细节:搜出来的配置不是一刀切。Ornith-397B 的最佳精度配置里,15 个全注意力层有两个干脆保留完整注意力,各层窗口从 256 到 4096 不等——\"每一层需要看多少上下文\"本身,就是被搜索出来的结论,而不是架构师的直觉。\n\n作者也自报了一个口径:\"据我们所知,这是首个在 100B+ 规模上评测的免训练稀疏注意力方法。\"听个参考即可,单源声明当结论用要打折。\n\n论文原文见 [arXiv:2611.06133](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.06133)。下次做长上下文容量规划,不妨先问每一层:你真的需要看完整 256k 吗?","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.06133","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"a1f919ce-5c76-4753-afef-c83817f447ab","en","Hybrid LLM Decoding 2.2x Faster at 256k, No Retraining","NSGA-II per-layer sparse attention gives hybrid LLMs up to 2.23x faster 256k decoding with no retraining; accuracy stays within 0.5pp on 21 datasets.","Hybrid LLM architectures were supposed to be the answer to long-context cost: replace most full-attention layers with linear attention, and per-step cost stops growing with context. But the few full-attention layers that remain still read the entire KV at every step, and at 256k tokens they become the new decoding bottleneck. A paper posted to arXiv on November 4 offers a fix that touches no weights: no retraining, no parameter edits — just a per-layer search over sparse-attention configurations. The result is a 2.23x decoding speedup at 256k context on Qwen3.5-122B-A10B, with accuracy intact across 21 long-context datasets.\n\n## The remaining full-attention layers are the long-context bill\n\nStandard self-attention makes every generated token more expensive than the previous one: quadratic time in sequence length, and a KV cache that grows with every token and every concurrent request. Hybrid models swap most layers for cheap attention, but how many full-attention layers survive is frozen into the weights at pretraining: Qwen3.5-27B keeps 16 out of 64, gpt-oss-120b keeps 18 out of 36, and Ornith-397B keeps 15 out of 60. Those layers read every key in the context at each step; they eat most of the decode budget on long tasks and cap how many concurrent requests a deployment can serve.\n\n## The method: sliding window + prefix + query-aware pages, searched per layer\n\nThe paper's combined sparse attention lets each full-attention layer see only three things: a sliding window of recent tokens, a pinned prefix, and a fixed budget of the most relevant tokens selected dynamically per query — Quest-style paged selection over the KV cache.\n\nThe real insight is \"per layer\". Window size and token budget are not hand-picked: an NSGA-II multi-objective search (accuracy vs estimated cost) finds a (window, budget) pair for every layer individually, with accuracy estimated on BABILong-qa1 at 64k plus the first 100 LongBench Qasper examples. The search is not free but bounded: 8.6–34.2 GPU-hours on A100s for the 4B–120B models, and 31.3–58.4 GPU-hours on a 4x GB200 node for the 122B and 397B models. The implementation plugs directly into vLLM.\n\nTwo honest caveats: prefill is untouched, and KV-cache memory is not reduced at all — the savings are in keys read per decoding step, that is, time, not VRAM.\n\n## Results and limits: a free 2x, and where you don't get it\n\nThe evaluation covers six models — Qwen3.5-4B\u002F27B\u002F122B-A10B, Ornith-1.0-397B, Gemma-4-31B, gpt-oss-120b — across 21 long-context datasets (BABILong qa1–qa5 at 64k plus the 16 English LongBench subsets), with the samples used during search excluded so all reported accuracy is held-out.\n\nOn every model, at least one configuration preserves baseline accuracy on every dataset, with averages within 0.5 percentage points. At 64k\u002F256k context, batch size 4: 1.26x\u002F2.22x on Qwen3.5-4B, 1.20x\u002F2.23x on Qwen3.5-122B-A10B, and 1.16x\u002F2.06x on Ornith-1.0-397B — the three best models all double their decoding speed at 256k.\n\nThe limits are stated plainly: at short context, every configuration is slower than the unmodified model. Dense models gain little — Qwen3.5-27B gets only 1.11x at 64k and Gemma-4-31B 1.22x at 128k, because at 64k the 27B model spends 19.4ms of its 47.1ms decode step reading weights, which attention tuning cannot fix. gpt-oss-120b manages only 1.16x at 128k, since the searched configuration kept most of its full-attention layers intact.\n\n## Why it matters\n\nCompare the retraining route: LoLCATs spent 454 H100 GPU-hours transferring attention on Llama 3.1 405B; Jet-Nemotron-2B hit a 12x speedup at 64k but paid with 600B training tokens and 18,328 H100 GPU-hours. A config search priced in GPU-hours means you can re-run it for every newly released model — a completely different amortization logic.\n\nOne more telling detail: the searched configs are not uniform. Ornith-397B's best-accuracy configuration leaves two of its 15 full-attention layers completely untouched, with per-layer windows ranging from 256 to 4096 — \"how much context each layer needs\" turns out to be a search result, not an architect's intuition.\n\nThe authors also state their own claim: \"To our knowledge, this is the first training-free sparse-attention method evaluated at the 100B+ scale.\" Treat it as a self-report, not a settled fact.\n\nFull paper: [arXiv:2611.06133](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.06133). Next time you plan long-context capacity, ask each layer: do you really need to read all 256k?","hybrid-llm-training-free-sparse-decoding","2026-11-05T13:08:09Z","2026-11-05T13:08:19.219675Z","2026-11-05T13:08:19.219687Z",true,"agent",43,[39,48],{"slug":40,"tag_slug":40,"title_zh":41,"title_en":42,"intro_zh":43,"intro_en":44,"id":45,"is_active":35,"created_at":46,"modified_at":47},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":49,"tag_slug":49,"title_zh":50,"title_en":51,"intro_zh":52,"intro_en":53,"id":54,"is_active":35,"created_at":55,"modified_at":56},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":58},[59,64,69,74,79,84],{"id":60,"title":61,"news_slug":62,"published_at":63},"19637bd9-df50-4ecb-87e5-2a4ab633ddff","SpecOPD并行采分支:投机解码训练吞吐16倍","specopd-branch-parallel-speculative-distillation","2026-11-05T23:20:00+00:00",{"id":65,"title":66,"news_slug":67,"published_at":68},"178aa5e5-2a4f-4a87-a97c-0da16295d96f","EMNLP 2026 OCGQuant:用通道配对治 NVFP4 陪葬误差,Qwen3-1.7B 接近 FP16","ocgquant-nvfp4-outlier-companion-grouping","2026-09-10T09:15:00+00:00",{"id":70,"title":71,"news_slug":72,"published_at":73},"77a567e5-2209-4981-882b-e2a3077a0012","先猜再听:全双工语音模型把投机解码搬到音频到来之前","speculative-listening-full-duplex-speech-decoding","2026-11-06T09:05:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"53d00d00-e4fb-41b0-a36a-37161203b514","KV缓存拼接复用丢题可至100%,修补要盯答案块","kv-cache-repair-budget-target-chunk","2026-11-05T15:09:02+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"d456840a-1a1d-412d-90ac-73ca5a5bbe10","MoE 路由器读出推理状态:砍规划省三成 token","moe-router-reasoning-behavior-steering","2026-11-05T09:21:56+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"64088ca6-c244-474c-93ce-cdbfe55de6d6","投机解码没用满?agent循环把加速比挖到10.87倍","specpilot-agentic-speculative-decoding","2026-11-04T13:15:17+00:00"]