[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-zyphra-zha-tree-indexed-subquadratic-attention":3,"topics-all":38,"news-related-f3edd5fd-5a5c-447f-8704-500856b96d09":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"f3edd5fd-5a5c-447f-8704-500856b96d09","ZHA:Zyphra 用树形索引把注意力成本压成次二次","Zyphra 的 ZHA 在 KV 缓存上搭一棵递归压缩的 key 摘要树,查询自顶向下路由,只取稀疏子集做精确注意力,注意力总成本随序列长度亚线性增长。主干冻结、4B token 蒸馏即可迁移,长上下文检索比 MiniMax 稀疏注意力高 14 分。但超长标度目前仍是模型外推,未经实测。","长上下文的账其实很好算。Attention 的 prefill 成本随序列长度平方增长，decode 阶段每生成一个 token 都要重读整份 KV cache，1M 上下文在工程上意味着显存和带宽同时撞墙。所以过去两年「次二次注意力」这个标签出了不少，但真正做到渐近次二次的很少——多数只是把常数因子砍小，或者把成本从注意力本身挪到了索引器里。\n\n[arXiv:2610.42112](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.42112) 上，Zyphra 研究团队给出了另一条路。\n\n## 注意力本来就稀疏，只是没人把它组织起来\n\n论文的第一件事不是造新算子，而是去量一个事实：自然语言序列上，注意力「真正有效」的状态是次线性的。团队在 Qwen3.6-35B-A3B 的全局注意力层上统计 top-p KV 子集的大小——即注意力概率累加到 p 所需的最小 key 集合——在六本长篇小说上随上下文长度增长的曲线呈次线性幂律。换句话说，序列越长，单个 query 真正需要看的 token 占比反而越小。\n\n但这里有个陷阱：如果某个 key 对当前 query 无关，它对后面的 query 可能至关重要。团队在 261,548 token 的序列上测量了「最近 W 个 query 的 top-p 子集并集」覆盖 KV 的比例，结果并集很快逼近全集。这直接否掉了「永久驱逐 KV」这条路——全量 KV 必须留着，只是每次按 query 取子集。\n\n## 用一棵压缩树替代扁平索引器\n\n细粒度稀疏注意力的常见做法是：轻量索引器给每个 KV 打分，只对 top-K 做精确注意力。问题在于索引器自己可能要扫全量 KV，成本被搬了个家，在接近 1M token 的区间，这类层仍有 60% 到 90% 的时间花在索引上。\n\nZHA 的做法是在 KV cache 之上搭一棵「递归压缩的 key 摘要树」：查询从树顶往下走，每一层按 top-p 路由挑出候选块，把子节点交给下一层，最终落到叶子块做精确注意力。树的深度逐级增加时，路由加最终注意力的总成本随序列长度亚线性增长，整体呈次二次（但仍是超线性）。\n\n关键工程性质是这个结构对几何形状不敏感。论文扫了 28 组配置（参考 query 数 R、块大小 C、深度 L），并在 p 从 0.7 到 0.99 的阈值上继续扫，发现只关注 10% 到 30% 的 key 时，检索精度对树形状相当稳健；在固定压缩率 C^L=4096 下，加深反而让有效检索能撑到更高稀疏度。团队还用一个 4 参数模型拟合了 RULER 精度，留一长度外推的 RMSE 只有 1.44 个百分点。\n\n## 实测：换索引器，不动主干\n\n这是 ZHA 最工程化的一面：主干权重完全冻结，只用 KL 蒸馏在几十亿 token 上训一个路由头。Qwen3.6-35B-A3B 上同时训了「深」（块大小 4、6 层）和「宽」（块大小 256、1 层）两种路由器，路由阈值取 0.8、0.9、0.95。\n\n结果：推理、数学和智能体编码基准上，两个路由器都与稠密注意力在噪声范围内持平。唯一例外是深路由器的 RULER-13——93.3% 对稠密的 96.0%，3 分的差距，而此时只看原 KV cache 的 1% 到 6%。\n\n对照更有意思。基准取的是 MiniMax 的稀疏注意力 [MSA](https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.13392)（块级稀疏加轻量 Index Branch，官方报告在 109B 模型上 1M 上下文时每 token 注意力算力降 28.4 倍，H800 上 prefill 14.2 倍、解码 7.6 倍）。团队给 MSA 的索引器用了 40B token 的蒸馏预热，是 ZHA 训练预算的 10 倍。但在 LongBench v2 上——MSA 的固定预算约等于上下文的 1%——ZHA 深路由器关注 1.1% 的 decode key，却高出 14 分。\n\n同一套方法也移植到了 Nemotron-3.5-Lightning-30B-A3B 上（该模型用 NoPE，没有显式位置编码），智能体编码上大体可比，RULER 上中等程度退化。有意思的是，NoPE 模型的 top-p 子集并不呈现那么规整的幂律——论文诚实地指出，幂律本身和位置编码有关。\n\n## 那个「百万上下文」是推出来的，不是跑出来的\n\n这是全篇最该被标注的地方。论文用 roofline 模型外推 ZHA 的延迟随序列长度的标度，结论是在 1M 之后会「大幅超过」稠密注意力与 MSA，对 MSA 的交叉点预测在 p=0.75 时约 0.5M token。\n\n但论文自己也写明：由于缺乏能在这种长度上运行的 LLM，这些超长标度没有经过实证验证；这篇工作也没有发布优化过的 kernel。所以「次二次」目前是一个有理论支撑的模型预测加一组 64K 到 256K 上的实测，外推那部分还要等硬件和模型同时跟上来才能兑现。论文提到的一个自证性观察是，据其所知这是首个在深度大于 1 时仍与稠密注意力保持接近水平的分层注意力方案——这个「首个」目前只有论文自述。\n\n## 所以呢\n\n真正值得记下的不是「下一个次二次注意力」，而是三个可复用的判断。\n\n一是注意力在自然语言上本来就次线性可压缩——这件事一旦被量出来，「上下文不够」就更像一道工程账，而不是能力上限。二是把索引器做成有层级的树而不是扁平的 top-K 打分，才能在固定预算下随长度继续省，否则省下的注意力成本会在索引器里还回去。三是迁移成本低到离谱：主干冻结、4B token、几十分之一的注意力暴露，就能换到接近稠密的效果。\n\n对做长上下文 Agent 的人，第三条可能最实用：不必等新模型发布，也不必重训，几十分之一的注意力预算就能把整份代码库或长文档塞进窗口。至于那个「数亿 token」的愿景，先当作有理论支撑的路线图，别当成已经跑通的数字。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.42112","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"c18f5074-eaa0-4bcc-b175-19eac5fffbb3","en","ZHA: Zyphra's tree indexer for subquadratic attention","Zyphra's ZHA puts a compressed key tree over the KV cache, making attention sublinear; 4B tokens beat MiniMax sparse attention by 14 points","The bill for long context is easy to work out. Attention prefill cost grows quadratically with sequence length, and at decode time every new token forces a re-read of the entire KV cache. A 1M-token context is, in engineering terms, a simultaneous collision of memory capacity and bandwidth. Plenty of papers have worn the \"subquadratic attention\" label over the past two years, but very few actually achieve asymptotic subquadratic behavior — most merely shrink the constant factor, or relocate the cost from attention itself into the indexer.\n\nA research team at Zyphra takes a different route in [arXiv:2610.42112](https:\u002F\u002Farxiv.org\u002Fabs\u002F2610.42112).\n\n## Attention was already sparse; nobody organized it\n\nThe first thing the paper does is not build a new operator. It measures a fact: on natural language sequences, the genuinely useful state of attention is sublinear. The team measured the size of the top-p KV subset in global attention layers of Qwen3.6-35B-A3B — the smallest set of keys whose attention probabilities sum to p — and found that across six long fiction books, that subset size grows as a sublinear power law with context length. Put differently: the longer the sequence, the smaller the fraction of tokens any single query actually needs to look at.\n\nThere is a trap here. A key that is irrelevant to the current query may matter enormously to a later one. On a 261,548-token sequence the team measured what fraction of the KV cache is covered by the union of top-p subsets across the W most recent queries, and found that union quickly approaches the entire cache. That directly rules out permanently evicting KV entries — the full cache has to stay, and each query simply retrieves its own subset.\n\n## A compressed tree instead of a flat indexer\n\nThe usual recipe for fine-grained sparse attention is simple: a lightweight indexer scores every KV pair, and exact attention runs only over the top-K. The problem is that the indexer itself may have to sweep the whole KV cache. The cost is moved rather than removed — at contexts approaching 1M tokens, 60% to 90% of the time in these layers is still spent on indexing.\n\nZHA builds a recursively compressed tree of key summaries on top of the KV cache. A query walks down from the top, each level routing to a candidate set of blocks via top-p selection and handing its children to the next level, until it lands on leaf blocks where exact attention runs. As the tree's depth grows iteratively, the total cost of routing plus final attention grows sublinearly in sequence length, yielding overall subquadratic (though still superlinear) scaling.\n\nThe key engineering property is that this structure is insensitive to geometry. The paper swept 28 configurations (reference query count R, chunk size C, tree depth L) and further swept routing thresholds from p = 0.7 to 0.99, finding retrieval accuracy fairly robust to tree shape when only 10% to 30% of keys are attended. At a fixed compression factor of C^L = 4096, going deeper actually lets effective retrieval hold up at higher sparsity. The team also fit RULER accuracy with a four-parameter model; leave-one-length-out extrapolation came out to an RMSE of just 1.44 percentage points.\n\n## Measured results: swap the indexer, leave the backbone alone\n\nThis is the most engineering-friendly aspect of ZHA: the backbone weights stay completely frozen, and only a router head is trained via KL distillation over a few billion tokens. On Qwen3.6-35B-A3B, the team trained both a \"deep\" router (chunk size 4, six tree levels) and a \"wide\" one (chunk size 256, one tree level), with routing thresholds of 0.8, 0.9 and 0.95.\n\nThe results: on reasoning, mathematical and agentic coding benchmarks, both routers matched dense attention within noise. The single exception was RULER-13 for the deep router — 93.3% against dense's 96.0%, a 3-point gap, and at that point only 1% to 6% of the original KV cache was being attended.\n\nThe comparison is the more interesting part. The baseline is MiniMax's sparse attention, [MSA](https:\u002F\u002Farxiv.org\u002Fabs\u002F2606.13392) — blockwise sparsity with a lightweight Index Branch, which the authors report reduces per-token attention compute by 28.4x at 1M context on a 109B model, with 14.2x prefill and 7.6x decoding wall-clock speedups on H800. The team gave MSA's indexer a 40B-token distillation warmup, 10x ZHA's training budget. But on LongBench v2 — where MSA's fixed budget amounts to roughly 1% of the context — the ZHA deep router attends a comparable 1.1% of decode keys and still scores 14 points higher.\n\nThe same method was also ported onto Nemotron-3.5-Lightning-30B-A3B, which uses NoPE and carries no explicit positional encoding. Agentic coding performance is broadly comparable, while RULER degrades moderately. Interestingly, the top-p subsets of the NoPE model do not exhibit such a regular power law — the paper credits that to the positional encoding itself.\n\n## The \"million-token context\" is extrapolated, not measured\n\nThis is the point in the paper that most needs flagging. The authors extrapolate ZHA's latency scaling with sequence length using a roofline model, concluding that beyond 1M tokens it would \"dramatically outperform\" both dense attention and MSA, with a predicted crossover against MSA around 0.5M tokens at p = 0.75.\n\nThe paper also states plainly that, for lack of LLMs able to operate at such lengths, these ultra-long scalings have not been empirically validated, and that this work does not release optimized kernels. So \"subquadratic\" currently amounts to a theoretically grounded model prediction plus a set of measurements between 64K and 256K; the extrapolated part will only be cashed out once hardware and models advance together. One self-referential observation the authors make is that, to their knowledge, this is the first hierarchical attention scheme to demonstrate convincing scaling to depths greater than 1 while staying close to dense attention — and that \"first\" currently rests on the paper's own assertion.\n\n## So what\n\nWhat is worth remembering is not \"yet another subquadratic attention,\" but three reusable judgments.\n\nFirst, attention is already sublinearly compressible on natural language — once that is measured, \"not enough context\" starts to look like an engineering bill rather than a capability ceiling. Second, the indexer has to be a hierarchy, not a flat top-K scoring pass, if savings are to keep compounding with length; otherwise the attention cost you saved comes back in the indexer. Third, the migration cost is remarkably low: frozen backbone, 4B tokens, a few percent of attention exposure, and you land close to dense performance.\n\nFor people building long-context agents, the third point is probably the most immediately useful. You do not have to wait for a new model release, and you do not have to retrain — a few percent of the attention budget can put an entire codebase or long document into the window. As for the vision of hundreds of millions of tokens, treat it as a roadmap with a theoretical foundation, not as a number that has already been demonstrated.","zyphra-zha-tree-indexed-subquadratic-attention","2026-11-02T05:21:18Z","2026-11-02T05:25:22.614772Z","2026-11-02T05:25:22.614784Z",true,"agent",451,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"abd8eb9f-c106-4899-bf24-b6bbed3da5e3","EPFL论文实测:注意力砍维度而非砍token,只算三成点积","hisa-hidden-dimension-sparse-attention-rope","2026-11-20T09:00:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"49972f78-da29-4041-8f1f-c57d9d8ceeac","MIT 一行代码修 MTP:八成梯度噪声被滤掉","soft-mtp-kl-gradient-recovery","2026-10-30T17:12:42+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a840ee24-8865-4038-986e-1398549feeb0","苹果论文重算循环大模型算力账:停止头说省一半,实测跑 7.6 轮","apple-halting-head-looped-lm-compute-audit","2026-10-26T03:05:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"282f3cb4-429c-4432-b632-5e7288192eb9","MassAlloc注意力:按质量分配算力,反传快3倍","massalloc-attention-mala","2026-09-29T21:08:59+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"0bdbdcd9-fbb5-4123-9854-57b8f28b385b","把注意力退回查字典:LEMA让KV缓存离开显存","lema-exact-match-attention","2026-09-25T15:15:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"b4f270b3-db43-4586-a0e5-a062320c6d1b","让模型自己声明看哪里:Declarative Attention 零训练砍 52% KV 读取","declarative-attention-kv-cache-declare","2026-09-03T23:07:03+00:00"]