[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-hisa-hidden-dimension-sparse-attention-rope":3,"topics-all":38,"news-related-abd8eb9f-c106-4899-bf24-b6bbed3da5e3":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"abd8eb9f-c106-4899-bf24-b6bbed3da5e3","EPFL论文实测:注意力砍维度而非砍token,只算三成点积","EPFL 与 ETH Zurich 的论文提出 HiSA:用 RoPE 旋转结构,在隐藏维度而非 token 维度上做稀疏。4K 上下文下只算 29.3% 到 34.2% 的 query-key 点积,验证损失反低于稠密注意力;16K 解码时 KV cache 少三分之一。论文给出可解释性证据与三处保留。","注意力把每个 query 和每个 key 做一次完整点积。主流做法是让 query 少看几个 key,滑窗、KV cache 淘汰、动态选块都走这条路。EPFL 与 ETH Zurich 的 Swiss Data Science Center 这篇论文换了个轴:不砍 key 的数量,砍点积本身的维度。按论文自己的实测,4K 上下文下算的 query-key 点积只有 29.3% 到 34.2%,验证损失不但没涨,还低于稠密注意力。\n\n## 一、RoPE 让点积可以被拆开\n\nRoPE 把位置编码塞进 query 和 key 的每一对二维频率。旋转会叠加,query 与 key 的两次旋转合成一次,内积里只留下相对距离 Δ = i - j。第 l 对的频率是 θ_l,注意力分数能精确分解成各频率对分数之和。\n\n频率高的对在几个 token 内就转过很多圈,它的分数对相对距离极其敏感;频率低的对在整个上下文里几乎不转,能承载长程依赖。\n\nHiSA 的规则只有一句:第 l 对的分数,只在它被转过的角度 Δθ_l 还低于角度预算 Φ 时才保留,默认 Φ = 90 度,也就是转过不到四分之一圈才留。快的几对只能看几个 token,慢的几对能看几千个,合起来形成一个 reach。论文的配方下 d = 64、Φ = 90 度,RoPE base 取 10^4 时 reach 是 11.8K token,取 2*10^4 是 23.1K。\n\n这个稀疏模式是静态且无参数的:只由位置和 RoPE 频率决定,不需要 router,不需要 top-k 选择,运行时也不打分挑 key。新引入的只有那个角度预算 Φ,投影、value 和 softmax 都没动。训练时用哪种稀疏,推理时就用哪种,不存在训练与推理不一致的问题。\n\n## 二、算得更少,损失反而更低\n\n论文从零预训练 SmolLM 架构的 135M 和 360M 模型,语料是 OpenWebText,预算约 20 token 每参数。4K 上下文下,135M 配 RoPE base 10^4:稠密验证损失 2.906,HiSA 2.880,点积只算了 29.3%;换 base 2*10^4 是 2.908 对 2.869,占比 34.2%;360M 上是 2.587 对 2.575,占比 29.3%。这个差距在上下文窗口的每个位置上都成立,不是只在开头几个 token 上省了损失。\n\n内核实现分两档。精确规则用自写 Triton kernel,4K 下算 29.3%,但训练比稠密 SDPA 慢 1.7 倍;块规则版本 Flash-HiSA 按 128 token 分块,占比 34.8%,吞吐 238k token 每秒,和稠密 FlashAttention-3 的 236k 基本持平,但两者不是同一个模型,训练和推理必须用同一套。\n\n推理端才是它省下来的地方。16K 上下文、batch 128 时,Flash-HiSA 13.0k token 每秒、KV cache 29.1 GiB;稠密是 9.2k token 每秒、45.0 GiB。吞吐高出 42%,缓存少了三分之一。即便用同一套内核、且一个频率对都不丢,Flash-HiSA 仍然快 13%,论文把这部分归给自己的 query tile 更小。\n\n已经训好的稠密模型也能改。135M 稠密模型不改任何权重直接切到 HiSA,验证损失从 2.906 变成 2.937;各自续训 1.35B token 后,稠密续训是 2.838,HiSA 续训是 2.836。\n\n## 三、模型自己就在压制那些分数\n\n论文还检查了稠密模型到底用不用那些被 HiSA 丢掉的频率对:拿真实分数部分和一个同角度、内容无关的对照分数比,看谁对注意力输出的影响更大。训练开始时两者差不多;训练 1B token 后,90 度线以内只有 2-5% 的格子低于对照,以外则被慢慢压下去,180 到 360 度区间的比例从 33% 涨到 48%,在 SmolLM-135M 上超过 180 度的部分有 87-95% 低于对照,而后者训练量约 600B token。相对影响力上,超过 720 度的部分从 4.6% 降到 2.1%,在 SmolLM-135M 上是 0.5%。\n\n也就是说,稠密模型花了一部分训练预算去压制这些分数,而且压制还在继续。HiSA 等于把这个过程直接写进架构。\n\n## 四、几盆该泼的冷水\n\n第一,实验规模只有 135M 和 360M,论文自己在结论里把放大到更大模型、下游任务基准、更好的 kernel 设计列为待办。\n\n第二,精确规则训练比稠密慢 1.7 倍,速度优势要靠块规则版本才拿到,而块规则是近似,每对最多多留 127 个 token。反向传播目前还没吃到稀疏的便宜,只有前向在省。\n\n第三,论文把分数按保留的频率对数量归一化做对照,验证损失 2.879,与 HiSA 的 2.880 持平,说明增益不是来自远处分数被隐式压小。但这个变体在长程复制任务上塌了:重复 2560 token 之前见过的 32 token 片段,它的 token 准确率只有 33%,HiSA 是 77%,稠密是 60%。被丢掉的维度并非在所有任务上都无所谓。\n\n第四,同一时期至少还有两篇论文在做同一件事。ATFlash 按 RoPE 波长给每个频率对开距离窗口,他们自己报告在 Qwen2.5-0.5B 和 Llama-3.2-3B 上剪掉 37-48% 的 query-key 内积项、top-1 匹配率保持 96-98%。WavePrune 把每个通道限制在第一个旋转周期内,他们自己报告 Qwen3-8B 的 HELMET 从 35.7 提到 40.0,32K 上下文下 prefill 与解码相对 FlashAttention-2 提速 1.15 倍和 1.24 倍。HiSA 论文列了这两篇,并把区别归纳为三点:训练预算更紧、90 度只有四分之一圈;规则落成了真正的 KV cache 缩减;补了上面那套可解释性分析。论文也点明,前两篇都跳过了计算和 key 读取,但都没真正缩小 KV cache。\n\n## 五、所以呢\n\n这批工作指向同一个判断:RoPE 的周期性在远距离上大部分是冗余的,一个频率对真正能编码精确相对位置的距离是有限的。同一个判断,三个团队从训练损失、推理内存、输出分布三个角度各自验证了一遍。\n\nHiSA 值得看的地方在于它把这件事做成了架构而不是推理技巧:改完之后可以直接当新模型训,也可以把已有的稠密模型续训转过去。对做长上下文部署的人来说,KV cache 少三分之一这条,可能比验证损失那点差距更有意思。\n\n需要提醒的是,这仍然是小模型上的结论。只算三成点积这件事,能不能在更大模型上、在真实的长文推理里原样兑现,目前还没有人验证过。论文见 https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.26363 ,仓库在 https:\u002F\u002Fgithub.com\u002Fambroise-b\u002Fhisa 。","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.26363","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"4f214978-cac1-4f39-aa4b-f92a0d0934b7","transformer",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"8df039a2-9cee-44e1-8396-dcfd49cbcd9e","en","EPFL: attention sparse in the hidden dimension, not tokens","EPFL's HiSA sparsifies attention in the hidden dimension, not over tokens: 29.3% of dot products at 4K, lower validation loss than dense.","Attention scores every query against every key with a full dot product. The standard way to make that cheaper is to show each query fewer keys: sliding windows, KV cache eviction, dynamic block selection. Hidden Sparse Attention, from EPFL and ETH Zurich, takes a different axis. It does not drop keys, it drops dimensions of the dot product itself. By the paper's own measurements, at 4K context it computes 29.3% of the query-key dot products and lands on a lower validation loss than dense attention.\n\n## 1. RoPE makes the dot product decomposable\n\nRoPE encodes position by rotating each two-dimensional frequency pair of the query and key. Rotations compose, so the query and key rotations collapse into one, and only the relative distance Delta = i - j survives in the inner product. The attention score decomposes exactly into a sum over frequency pairs.\n\nHigh-frequency pairs complete many turns within a few tokens, so their score parts are extremely sensitive to the exact relative distance. Low-frequency pairs barely rotate across the whole context, and they carry the long-range dependency.\n\nThe HiSA rule is one sentence: the score part of pair l is kept only while the rotation it induces, Delta theta_l, stays below an angle budget Phi. The default is Phi = 90 degrees, meaning a pair contributes while RoPE has turned it by less than a quarter of a turn. The fastest pairs then see only a few tokens, the slowest several thousand, and together they define a reach. With d = 64 and Phi = 90, the reach is 11.8K tokens at RoPE base 10^4, and 23.1K at 2 times 10^4.\n\nThe sparsity pattern is static and parameter-free. It depends only on the distance and on the RoPE frequencies, not on the content of the queries and keys, so there is no router and no top-k selection, and nothing at runtime scores and picks keys. The only new quantity is the hyperparameter Phi; the projections, the values and the softmax are unchanged. Because the pattern is fixed by positions, the model is trained with exactly the pattern it uses at inference.\n\n## 2. Less compute, lower loss\n\nThe authors pretrain SmolLM-architecture models of 135M and 360M parameters from scratch on OpenWebText, with a budget of about 20 tokens per parameter. At 4K context, with 135M and RoPE base 10^4, dense attention reaches a validation loss of 2.906 and HiSA 2.880, while computing 29.3% of the dot products. At base 2 times 10^4 the pair is 2.908 against 2.869, at 34.2% of the work. At 360M the numbers are 2.587 against 2.575, again at 29.3%. The gap holds at every position of the predicted token in the context window, so it is not an artifact of the first few tokens.\n\nThe kernels come in two flavours. The exact rule uses a hand-written Triton kernel and computes 29.3% at 4K, but trains 1.7 times slower than dense attention with SDPA. The block rule, Flash-HiSA, groups positions into blocks of 128 tokens and computes 34.8%, running at 238k tokens per second against 236k for dense FlashAttention-3. The two rules are different models: train and run a model with the same one.\n\nInference is where the saving actually shows up. At 16K context and batch 128, Flash-HiSA reaches 13.0k tokens per second with a KV cache of 29.1 GiB, against 9.2k tokens per second and 45.0 GiB for dense. That is 42% more throughput and a cache a third smaller. Even with the same kernel and no pair dropped at all, Flash-HiSA is still 13% faster, which the paper attributes to its smaller query tiles.\n\nExisting dense models can be converted too. The dense 135M model is switched to HiSA without changing any weight, taking its validation loss from 2.906 to 2.937; after each continues for 1.35B tokens, dense reaches 2.838 and HiSA 2.836.\n\n## 3. Dense attention is already suppressing those parts\n\nThe paper also asks whether dense models even use the frequency pairs that HiSA drops. The method compares a real score part against a control score part with the same rotation but unrelated content, and measures which one moves the attention output more. At initialization the two are close. After 1B tokens, only 2-5% of the cells below 90 degrees fall below the control, while beyond that line the real parts are gradually pushed down: in the 180 to 360 degree band the share grows from 33% to 48%, and in SmolLM-135M, trained on about 600B tokens, 87-95% of the cells beyond 180 degrees fall below the control. In terms of relative influence on head outputs, parts beyond 720 degrees drop from 4.6% after 1B tokens to 2.1% at the end of training, and to 0.5% in SmolLM-135M.\n\nSo a dense model spends part of its training suppressing score parts with a large rotation, and the suppression is still progressing at the end of the runs. HiSA writes that behaviour into the architecture.\n\n## 4. Four cold-water notes\n\nFirst, the experiments cover only 135M and 360M models. The paper itself lists scaling to bigger models, downstream task benchmarking and better kernel design as open questions.\n\nSecond, the exact rule trains 1.7 times slower than dense attention, so the speed argument depends on the block version, which is an approximation that keeps up to 127 more tokens per pair. The backward pass does not yet benefit from the sparsity, so only the forward pass saves anything.\n\nThird, the paper trains a control variant that normalises each score by the number of pairs it keeps. It reaches a validation loss of 2.879, on par with HiSA at 2.880, so the gain does not come from the shrinking scale of distant scores. But the variant collapses on a long-range copy task, repeating a 32-token span seen 2560 tokens earlier: its token accuracy is 33%, against 77% for HiSA and 60% for dense attention. The dropped dimensions are not irrelevant on every task.\n\nFourth, at least two other papers were working on the same idea around the same time. ATFlash opens a distance window per RoPE wavelength and reports pruning 37-48% of the query-key inner-product terms on Qwen2.5-0.5B and Llama-3.2-3B, with the top-1 match rate staying at 96-98%. WavePrune restricts each channel to its first rotation period and reports the HELMET score on Qwen3-8B rising from 35.7 to 40.0, with 1.15x prefill and 1.24x decoding speedups over FlashAttention-2 at 32K context. The HiSA paper lists both, and frames its own differences in three ways: it trains with much tighter budgets, a quarter period at 90 degrees; it turns the rule into an actual KV-cache reduction; and it adds the interpretability analysis described above. The paper also notes that the other two skip computation and key reads but neither reduces the KV cache, since every key and value stays in memory.\n\n## 5. So what\n\nAll of this points at one conclusion: the periodicity of RoPE is largely redundant at long range, because the distance over which a frequency pair can encode an exact relative position is finite. Three groups arrived at it from three directions, training loss, inference memory and output distribution.\n\nWhat makes HiSA worth reading is that it turns this into an architecture rather than an inference trick. You can train a new model with it, or convert an existing dense model by continued training. For anyone serving long contexts, a KV cache roughly a third smaller is probably more interesting than the small difference in validation loss.\n\nThe caveat is that all of this comes from small models. Whether computing 30% of the dot product still holds on much larger models, in real long-document inference, has not been shown yet. The paper is at https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.26363 and the code repository at https:\u002F\u002Fgithub.com\u002Fambroise-b\u002Fhisa .","hisa-hidden-dimension-sparse-attention-rope","2026-11-20T09:00:00Z","2026-11-20T09:15:36.567652Z","2026-11-20T09:15:36.567663Z",true,"agent",66,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"f3edd5fd-5a5c-447f-8704-500856b96d09","ZHA:Zyphra 用树形索引把注意力成本压成次二次","zyphra-zha-tree-indexed-subquadratic-attention","2026-11-02T05:21:18+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"49972f78-da29-4041-8f1f-c57d9d8ceeac","MIT 一行代码修 MTP:八成梯度噪声被滤掉","soft-mtp-kl-gradient-recovery","2026-10-30T17:12:42+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a840ee24-8865-4038-986e-1398549feeb0","苹果论文重算循环大模型算力账:停止头说省一半,实测跑 7.6 轮","apple-halting-head-looped-lm-compute-audit","2026-10-26T03:05:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"282f3cb4-429c-4432-b632-5e7288192eb9","MassAlloc注意力:按质量分配算力,反传快3倍","massalloc-attention-mala","2026-09-29T21:08:59+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"0bdbdcd9-fbb5-4123-9854-57b8f28b385b","把注意力退回查字典:LEMA让KV缓存离开显存","lema-exact-match-attention","2026-09-25T15:15:00+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"b4f270b3-db43-4586-a0e5-a062320c6d1b","让模型自己声明看哪里:Declarative Attention 零训练砍 52% KV 读取","declarative-attention-kv-cache-declare","2026-09-03T23:07:03+00:00"]