[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-hyquant-hybrid-precision-attention-quantization":3,"topics-all":35,"news-related-49c90242-7793-46c7-961a-8a39e608e23d":54},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":21,"news_slug":28,"published_at":29,"created_at":30,"modified_at":31,"is_published":32,"publish_type":33,"image_url":14,"view_count":34},"49c90242-7793-46c7-961a-8a39e608e23d","竖线 token 才是量化命门：HyQuant 混合精度让 4-bit 注意力几乎无损","上海交大 HyQuant 论文发现注意力误差集中在少数\"竖线\"token——top 5% key 加 128 token 窗口覆盖 82-86% 注意力质量。关键 token 留 FP16、其余压 4-bit 后，LongBench 反超全精度 FA2，decode kernel 最高快 3.58 倍。","低比特量化最容易翻车的地方，不是权重，是注意力。上海交大团队 8 月底放上 arXiv、已被 EMNLP 2026 Main 录用的 HyQuant 论文，把翻车原因指到一个很具体的结构上：注意力图里的「竖线 token」。理解了这个结构，4-bit 的注意力几乎可以无损压缩。\n\n## 竖线：误差都挤在少数 token 上\n\n论文作者在 Hugging Face 论文页评论区补充了一组关键观察：在 Qwen3、Llama-3、Gemma 4、Qwen3.5 上，注意力图持续存在「竖线」——少数 key 位置被几乎所有 query 关注。量化这些位置的误差会被放大到整个序列；反过来，top 5% 的 key 位置加上一个 128 token 的局部滑动窗口，就能覆盖 82-86% 的注意力质量，而识别这些竖线只需 3-5% 的运行时开销。\n\n这组数字解释了为什么低比特 KV-cache 量化常常伤到长上下文推理：误差不是均匀分布的，而是集中砸在少数高关注 token 上。论文摘要也点明，现有方法主要靠 smoothing 技术处理离群值——这类平均主义方案抹不掉结构性集中的误差。\n\n## 混合精度：关键的留 FP16，其余压进 4-bit\n\nHyQuant 的设计顺着这个结构来：竖线 token 和局部窗口保留 FP16，其余注意力状态压成 K4V4 的 4-bit 格式。prefill 阶段用混合精度量化注意力算子；decode 阶段把同一原则用于 KV-cache 压缩，并将 KV 反量化与注意力计算融合，避免 KV cache 物化，省下显存与访存带宽。作者称这套设计\"极其简单\"，在多任务、多模型、多数据集上保持接近无损的精度。\n\n## 跑分与边界\n\n论文自报的单张 H100 数字：\n\n- LongBench 平均分 45.04，高于 FlashAttention-2 全精度的 44.59（Qwen3-8B thinking 模式）；作为对照，KIVI、SageAttention、KVTuner 只有 37.7-40.5\n- decode kernel 在 32K 上下文比 FlashAttention-2 快至 3.58 倍；端到端 decode 提速 1.04-1.17 倍，而 KIVI\u002FKVTuner 端到端反而比 FA2 慢（0.69-0.80 倍）\n- batch 16 搭配 32K prefix 时，它是唯一还能跑的方法（231.6 tok\u002Fs），FA2、KIVI、KVTuner 全部 OOM\n- 上述结论在 Qwen3-32B、Llama-3.1-8B、GLM-4-9B 上复现\n\n边界要讲清楚：以上均为论文与作者自报数字，代码已在 GitHub 开源，第三方复现还有待社区验证。\n\n## 所以呢\n\n「压到几 bit」是量化工程的老问题，「误差集中在哪」是 HyQuant 换的新问题。注意力天然存在结构性稀疏——竖线加局部窗口——意味着精度预算应该按注意力质量分配，而不是按比特均匀分配。对做长上下文推理降本的团队，这是条比全模型量化更立竿见影的路：注意力与 KV-cache 恰好是长上下文显存的大头，batch 16 + 32K 下唯一存活的实测也说明它解决的是真实容量瓶颈而非纸面速度。EMNLP 2026 Main 的录用说明这条路线已过同行评审第一关，接下来值得盯的是社区在更多模型家族上的复现结果。\n\n参考：arXiv:2608.27875（https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.27875）；代码：github.com\u002Fjerrysfls\u002FHyQuant","https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.27875","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":19,"name":20,"slug":20,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[22],{"id":23,"lang":24,"title":25,"summary":26,"content":27},"02e046b6-27de-482f-933e-720d945f21b4","en","HyQuant: hybrid precision makes 4-bit attention nearly lossless","HyQuant keeps 5% of attention tokens in FP16, quantizes the rest to 4-bit: LongBench tops FA2, decode kernel up to 3.58x faster at 32K.","Low-bit quantization breaks attention first, not weights. HyQuant, a Shanghai Jiao Tong University paper accepted to EMNLP 2026 Main, pinpoints why: attention maps across Qwen3, Llama-3, Gemma 4 and Qwen3.5 show persistent \"vertical lines\" — a small set of key positions that nearly every query attends to. Quantizing those positions amplifies error across the whole sequence.\n\n## Vertical lines: the error is not uniform\n\nThe paper's authors added a key set of numbers in the Hugging Face paper-page comment section: the top 5% of key positions plus a 128-token local sliding window capture 82-86% of total attention mass. Identifying those vertical-line tokens costs only 3-5% of runtime. This explains why low-bit KV-cache quantization tends to hurt long-context reasoning — error concentrates on a few heavily attended tokens instead of spreading evenly, and smoothing-based outlier handling cannot remove structurally concentrated error.\n\n## Hybrid precision: keep the critical few in FP16, quantize the rest to 4-bit\n\nHyQuant's design follows that structure directly. Vertical-line tokens and the local window stay in FP16; the remaining attention states are quantized to 4-bit K4V4. In the prefill stage it uses a hybrid-precision quantized attention operator. In the decode stage the same principle applies to KV-cache compression, fusing KV dequantization with attention computation so the KV cache is never materialized — saving memory and bandwidth. The authors describe the design as extremely simple, maintaining near-lossless accuracy across diverse tasks, models and datasets.\n\n## Reported numbers and their limits\n\nSelf-reported single-H100 results from the paper:\n\n- LongBench average 45.04 versus 44.59 for full-precision FlashAttention-2 (Qwen3-8B, thinking mode); KIVI, SageAttention and KVTuner land at 37.7-40.5\n- Decode kernel up to 3.58x faster than FlashAttention-2 at 32K context; end-to-end decode 1.04-1.17x faster, while KIVI and KVTuner end up slower than FA2 (0.69-0.80x)\n- At batch 16 with a 32K prefix, it is the only method that still runs (231.6 tok\u002Fs) — FA2, KIVI and KVTuner all run out of memory\n- Results hold on Qwen3-32B, Llama-3.1-8B and GLM-4-9B\n\nCaveats in plain terms: all numbers above come from the paper and its authors; the code is open-sourced on GitHub, and third-party replication is still pending.\n\n## So what\n\n\"How many bits\" is the old quantization question; \"where the error concentrates\" is the one HyQuant asks instead. Attention has natural structural sparsity — vertical lines plus a local window — which means precision budget should follow attention mass, not be spread evenly across bits. For teams cutting long-context inference cost, this is a more direct lever than whole-model quantization: attention and KV-cache dominate long-context memory, and the batch-16 survival test shows it addresses a real capacity bottleneck rather than paper speed. EMNLP 2026 Main acceptance means the route has cleared peer review's first gate; the next thing to watch is community replication across more model families.\n\nReference: arXiv:2608.27875 (https:\u002F\u002Farxiv.org\u002Fabs\u002F2608.27875); code: github.com\u002Fjerrysfls\u002FHyQuant","hyquant-hybrid-precision-attention-quantization","2026-09-13T17:06:39Z","2026-09-13T17:06:42.456477Z","2026-09-13T17:06:42.456486Z",true,"agent",55,[36,45],{"slug":37,"tag_slug":37,"title_zh":38,"title_en":39,"intro_zh":40,"intro_en":41,"id":42,"is_active":32,"created_at":43,"modified_at":44},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":46,"tag_slug":46,"title_zh":47,"title_en":48,"intro_zh":49,"intro_en":50,"id":51,"is_active":32,"created_at":52,"modified_at":53},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":55},[56,61,66,71,76,81],{"id":57,"title":58,"news_slug":59,"published_at":60},"2266cea6-06f1-4932-8905-1bc3f2e5a8c0","Meta FAIR 字节蒸馏研究:End-Of-Token 渐近反超 token 蒸馏 4%,数据只需 1\u002F6","meta-fair-byte-distillation-token-ceiling-2026-09","2026-09-15T02:00:00+00:00",{"id":62,"title":63,"news_slug":64,"published_at":65},"e91b3add-4f1d-48d3-ab7a-bd2c6e8c1765","QuIP 崩、OPTQ 降级:Kashin-DCT 在 4-bit 量化压力测试里活了下来","kashin-dct-2bit-llm-quantization","2026-09-12T15:10:00+00:00",{"id":67,"title":68,"news_slug":69,"published_at":70},"178aa5e5-2a4f-4a87-a97c-0da16295d96f","EMNLP 2026 OCGQuant:用通道配对治 NVFP4 陪葬误差,Qwen3-1.7B 接近 FP16","ocgquant-nvfp4-outlier-companion-grouping","2026-09-10T09:15:00+00:00",{"id":72,"title":73,"news_slug":74,"published_at":75},"6d60f9ba-4866-4520-9f12-d955e37f8472","Gated DeltaNet 全压 4-bit 没掉点:一篇论文拆掉混合 LLM 的量化禁忌","gated-deltanet-nvfp4-full-4bit","2026-09-04T15:08:02+00:00",{"id":77,"title":78,"news_slug":79,"published_at":80},"48e1c261-a40a-4c71-9cba-450a459e6ad3","4-bit 模型反超全精度:QAH 把量化从性能税变成第二次蒸馏","quantization-aware-healing-hypernova-60b","2026-08-25T17:20:00+00:00",{"id":82,"title":83,"news_slug":84,"published_at":85},"4147f71b-eaa7-4d39-91cf-c2c572105e7f","FlashPrefill V2:128K 长文本 prefill 提速 47 倍,块稀疏注意力走进生产框架","flashprefill-v2-block-sparse-prefill","2026-08-21T19:10:00+00:00"]