[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-ocgquant-nvfp4-outlier-companion-grouping":3,"topics-all":41,"news-related-178aa5e5-2a4f-4a87-a97c-0da16295d96f":60},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":27,"news_slug":34,"published_at":35,"created_at":36,"modified_at":37,"is_published":38,"publish_type":39,"image_url":14,"view_count":40},"178aa5e5-2a4f-4a87-a97c-0da16295d96f","EMNLP 2026 OCGQuant:用通道配对治 NVFP4 陪葬误差,Qwen3-1.7B 接近 FP16","华南理工 + Intellifusion 提出 OCGQuant,把 NVFP4 量化里的\"陪葬误差\"用 channel permutation 治好,Qwen3-1.7B 的 PPL 从 19.25 拉回 16.71,几乎贴 FP16,且推理开销与 RTN 持平。","## 背景：NVFP4 把\"低比特\"从研究推到产线\n\n最近一两年 LLM 推理成本被一步步往下压，INT4\u002FINT8 之后，业界开始把目光投向更激进的 4-bit 浮点。NVFP4 是 NVIDIA 主推的硬件原生 FP4 微缩格式——每个连续 16 个元素共用一个 FP8 scale，再加上一个张量级的 FP32 scale——比 INT4 表达力更细，比 FP16 又省一档带宽。问题是：真实 LLM 的激活值里，少数几个 channel 的数值能比中位数高出几个量级，量化时这些离群点直接决定 block scale，让同一 block 里\"陪葬\"的普通值被丢进 FP4 的粗糙格点，损失精度的就是这个陪葬过程。\n\n华南理工和 Intellifusion 的团队（arXiv 2609.00066，EMNLP 2026 主会）给这个问题起了一个名字：**Collateral Quantization Error**（陪葬量化误差），并提出了一种不需要额外算子、几乎不增加推理开销的 PTQ 方案：OCGQuant（Outlier-Companion Grouping）。开源代码在 `https:\u002F\u002Fgithub.com\u002FEshamont\u002FOCGQuant`，文章已挂在 [arXiv:2609.00066](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.00066) 上。\n\n## 核心思路：把\"陪葬者\"换成安静的通道\n\n传统方法要么走混合精度（LLM.int8、Atom，用 INT8 单独兜住离群点），要么走 Hadamard 旋转（QuaRot 等）把离群能量摊平，要么加冗余通道做残差补偿（ARCQuant）。这些方法在 INT 量化上有效，但对 NVFP4 的两点特性水土不服：FP4-E2M1 的非均匀格点不能简单用 uniform step 套；block-level scale 只覆盖 16 个 channel，意味着 channel 顺序直接决定 block 组成。\n\nOCGQuant 的切入点是后者：既然 block 是按 channel 顺序切片的，那就**重新排 channel**。具体做法是：\n\n- 在校准集上跑一遍前向，统计每个输入 channel 的 RMS；\n- 按 RMS 从大到小排队，**每个高 RMS 的离群 channel 都从剩下未分配的 channel 里，挑 15 个 RMS 最低的\"安静\"channel 配成一组**；\n- 同样的 channel permutation 套到权重上，离线用 GPTQ 补一遍 weight 端的量化误差。\n\n这个打法的直觉很朴素：低 RMS 通道值本来就接近 0，而 FP4-E2M1 在小数值区间的格点密度又相对较高（用作者 Lemma 1 的话，dFP4(z)² ≤ ½z² + 1\u002F32），所以\"安静者\"在离群者撑大的 scale 下，被量化的损失是可控的。\n\n论文还给了一个\"为什么挑低 RMS 作伴\"的形式化证明：当离群 candidate 固定时，bad event 的上界里依赖 companion 的那一项就是 ∑rf²，于是选 G−1 个最小 RMS 通道确实最优。\n\n## 数字：Qwen3-1.7B 几乎贴着 FP16\n\n实验在 Llama3.1-8B、Llama3.2-3B、Qwen3-8B、Qwen3-1.7B 四个模型上跑，perplexity 拿 WikiText-2 测，下游任务取六个 zero-shot 平均，摘录自论文 Table 7：\n\n| 模型 | FP16 | NVFP4 RTN | OCGQuant |\n|---|---|---|---|\n| Llama3.2-3B | 7.81 | 8.62 | **8.37** |\n| Llama3.1-8B | 6.24 | 6.95 | **6.75** |\n| Qwen3-1.7B | 16.67 | 19.25 | **16.71** |\n| Qwen3-8B | 9.72 | 10.07 | **9.90** |\n\n最亮眼的是 Qwen3-1.7B——RTN 把 PPL 从 16.67 顶到 19.25，OCGQuant 拉回 **16.71**，几乎贴着 FP16。同时相比同场竞技的 NVFP4 方案 ARCQuant，OCGQuant 在四个模型上都更低（ARCQuant 在 Llama3.1-8B 6.87，Qwen3-8B 10.35）。论文 Appendix 里还跑了 RPTQ-style（把 INT 时代的 channel clustering 直接搬到 NVFP4），结果在 Qwen3-1.7B 上甚至比 RTN 还差，说明为 INT 设计的重排规则在非均匀 FP4 格点上确实有副作用。\n\n## 推理成本：prefill 接近 RTN，memory 不输 RTN\n\n- Prefill 吞吐（Llama3.1-8B，seq=2048，batch=16）：FP16 11556 tokens\u002Fs，RTN 26838，**OCGQuant 26439**，接近 RTN；ARCQuant 只有 22784。论文里写\"OCGQuant achieves up to 2.29× prefill speedup\"对比的是 FP16 基准。\n- 解码显存（同模型同 batch=1）：FP16 15.33 GB，RTN 6.02 GB，**OCGQuant 6.02 GB**；ARCQuant 6.86 GB，多出来的是它的残差通道。\n\n换句话说，ARCQuant 用\"多加一条残差路径\"换精度，OCGQuant 用\"重排 channel\"换精度，后者在前向路径上几乎没新东西，所以**显存贴着 RTN、prefill 速度贴着 RTN**，在产线部署里这两个指标才是钱。\n\n## 离线代价与可重复性\n\n- RMS profiling：43.52 s（Llama3.1-8B）\u002F 45.04 s（Qwen3-8B）\n- Companion 选择：0.55 s \u002F 0.89 s\n- Reorder：1.97 s \u002F 4.52 s\n- GPTQ 重建：248.55 s \u002F 250.44 s\n- 合计 ~5 分钟 per 8B 模型\n\n校准数据集和样本量的 robustness 也不错：在 WikiText-2 \u002F C4 \u002F Pile 三种校准集、64\u002F128\u002F256\u002F512 四档样本量下，Qwen3-8B 的六任务平均准确率标准差在 0.10 上下，最大 0.24。**128 个校准样本就够**，多了也没更多收益。\n\n## 一点看法\n\nNVFP4 之所以被关注，是因为 Blackwell 之后的硬件把 4-bit 微缩变成原生算子——这件事已经在发生，所以软侧的 PTQ 方法最近半年密集冒头。OCGQuant 的价值不在\"它多了一个 SOTA\"，而在\"它给出了一条**几乎不增加推理开销**的工程化路径\"：重排是离线算的，运行时 channel permutation 是 fixed 的（论文里给了一个 fused CUDA kernel + contiguous reorder-segment fast path），不需要新算子、不需要混合精度、不需要新硬件。\n\n后续值得关注的两件事：一是作者团队隶属的 Intellifusion 和 SCUT（华南理工）的组合，意味着国内在 NVFP4 工程化上正与 NVIDIA 路线同步推进；二是 channel-locality + cross-token persistence 的发现（论文 Figure 8）很硬核——离群 channel 是\"贴着 channel 走的窄带\"，这意味着以后针对 NVFP4 的 outlier-aware PTQ 不必在每 token 上动态调，可以一次校准、长效使用。\n\n一句话总结：**NVFP4 的\"陪葬误差\"被一张 permutation 表治了**，下一个问题是：当 8-bit 都不肯用、必须压到 FP4 的边缘推理场景，这套 channel-grouping 思路能不能跟着 NPU\u002FTPU 一起卷过去。\n\n参考资料：\n- 论文: \u003Chttps:\u002F\u002Farxiv.org\u002Fabs\u002F2609.00066>\n- 代码: \u003Chttps:\u002F\u002Fgithub.com\u002FEshamont\u002FOCGQuant>\n- HTML 版（含完整 Table 7\u002F8\u002F9\u002F10、Figure 6\u002F7\u002F8、Theorem 1 证明）: \u003Chttps:\u002F\u002Farxiv.org\u002Fhtml\u002F2609.00066v1>","https:\u002F\u002Farxiv.org\u002Fhtml\u002F2609.00066v1","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21,24],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",{"id":25,"name":26,"slug":26,"description":14,"color":14},"c187600e-804c-4697-b828-1e4330e0eb10","qwen",[28],{"id":29,"lang":30,"title":31,"summary":32,"content":33},"80d19c97-0206-40b8-975a-78f5773107d9","en","EMNLP 2026: OCGQuant tames NVFP4 outliers, Qwen3-1.7B nears FP16","SCUT + Intellifusion propose OCGQuant, a channel-permutation PTQ that fixes NVFP4 \"collateral error\" with no extra inference path. On Qwen3-1.7B, PPL drops from 19.25 (RTN) to 16.71, near FP16 (16.67).","## Background: NVFP4 pushes \"low-bit\" from research to production\n\nOver the past year or two, LLM inference cost has been pushed down step by step. After INT4\u002FINT8, the industry has started looking at more aggressive 4-bit floating point. NVFP4 is NVIDIA's hardware-native FP4 microscaling format — every contiguous 16 elements share one FP8 scale, plus a tensor-level FP32 scale — which gives finer expressiveness than INT4 and saves another band of bandwidth vs FP16. The problem: in real LLM activations, a small number of channels can have values several orders of magnitude larger than the median. At quantization time, those outliers directly set the block scale, forcing the \"collateral\" ordinary values in the same block into the coarse grid of FP4 — and that is where the accuracy is lost.\n\nA team from South China University of Technology and Intellifusion (arXiv 2609.00066, EMNLP 2026 main conference) gave this problem a name: **Collateral Quantization Error**, and proposed a PTQ scheme that does not need extra operators and barely adds any inference overhead: OCGQuant (Outlier-Companion Grouping). The code is open-sourced at `https:\u002F\u002Fgithub.com\u002FEshamont\u002FOCGQuant`, and the paper is on [arXiv:2609.00066](https:\u002F\u002Farxiv.org\u002Fabs\u002F2609.00066).\n\n## Core idea: replace the \"collateral damage\" with quiet channels\n\nConventional methods either go mixed-precision (LLM.int8, Atom, using INT8 to catch outliers), Hadamard rotation (QuaRot, etc.) to spread outlier energy, or augmented residual channels for residual compensation (ARCQuant). These work for INT quantization, but do not transfer cleanly to NVFP4 for two reasons: the FP4-E2M1 non-uniform grid cannot be modeled by a single uniform step, and the block-level scale only covers 16 channels, which means the channel order directly determines block composition.\n\nOCGQuant's lever is the latter: since the block is sliced by channel order, **reorder the channels**. The procedure:\n\n- Run one forward pass on a calibration set, record each input channel's RMS;\n- Sort by RMS descending; **pair each high-RMS outlier channel with the 15 lowest-RMS \"quiet\" channels from the unassigned pool**;\n- Apply the same channel permutation to the weights, and use GPTQ offline to fix the weight-side quantization error in the reordered space.\n\nThe intuition is plain: low-RMS channels are already close to 0, and FP4-E2M1's grid is denser in the small-magnitude region (the authors' Lemma 1: dFP4(z)² ≤ ½z² + 1\u002F32), so under the outlier-enlarged scale, the loss when quantizing \"quiet neighbors\" is bounded.\n\nThe paper also gives a formal proof for \"why pick low RMS as companions\": when the outlier candidate is fixed, the only companion-dependent term in the bad-event upper bound is ∑rf², so picking the G−1 smallest-RMS channels is indeed optimal.\n\n## Numbers: Qwen3-1.7B nearly ties FP16\n\nThe experiments run on Llama3.1-8B, Llama3.2-3B, Qwen3-8B, Qwen3-1.7B, with perplexity measured on WikiText-2 and downstream accuracy averaged across six zero-shot tasks (excerpted from Table 7):\n\n| Model | FP16 | NVFP4 RTN | OCGQuant |\n|---|---|---|---|\n| Llama3.2-3B | 7.81 | 8.62 | **8.37** |\n| Llama3.1-8B | 6.24 | 6.95 | **6.75** |\n| Qwen3-1.7B | 16.67 | 19.25 | **16.71** |\n| Qwen3-8B | 9.72 | 10.07 | **9.90** |\n\nThe brightest result is Qwen3-1.7B: RTN inflates PPL from 16.67 to 19.25, OCGQuant pulls it back to **16.71**, essentially tied with FP16. Compared with the same-generation NVFP4 method ARCQuant, OCGQuant is lower on all four models (ARCQuant: Llama3.1-8B 6.87, Qwen3-8B 10.35). The Appendix also re-runs RPTQ-style (channel clustering ported from INT to NVFP4), which actually degrades below RTN on Qwen3-1.7B — proof that reordering rules designed for INT uniform grids have side effects on non-uniform FP4.\n\n## Inference cost: prefill close to RTN, memory matches RTN\n\n- Prefill throughput (Llama3.1-8B, seq=2048, batch=16): FP16 11556 tokens\u002Fs, RTN 26838, **OCGQuant 26439**, close to RTN; ARCQuant only 22784. The paper reports \"OCGQuant achieves up to 2.29× prefill speedup\" relative to FP16.\n- Decoding memory (same model, batch=1): FP16 15.33 GB, RTN 6.02 GB, **OCGQuant 6.02 GB**; ARCQuant 6.86 GB, the extra is its residual channels.\n\nIn other words, ARCQuant trades \"an extra residual path\" for accuracy, while OCGQuant trades \"a channel reorder\" for accuracy. The latter adds nothing new to the forward path, so **memory tracks RTN, prefill speed tracks RTN** — and on production deployments, those two metrics are the bill.\n\n## Offline cost and reproducibility\n\n- RMS profiling: 43.52 s (Llama3.1-8B) \u002F 45.04 s (Qwen3-8B)\n- Companion selection: 0.55 s \u002F 0.89 s\n- Reorder: 1.97 s \u002F 4.52 s\n- GPTQ reconstruction: 248.55 s \u002F 250.44 s\n- Total: ~5 minutes per 8B model\n\nRobustness across calibration sets and sample sizes is also good: across three calibration sets (WikiText-2 \u002F C4 \u002F Pile) and four sample sizes (64\u002F128\u002F256\u002F512), the standard deviation of six-task average accuracy on Qwen3-8B is around 0.10, max 0.24. **128 calibration samples is enough**; going larger does not buy more.\n\n## A few notes\n\nNVFP4 is on the radar because post-Blackwell hardware turns 4-bit microscaling into a native operator — that is already happening, which is why NVFP4 PTQ methods have been popping up densely in the past six months. OCGQuant's value is not \"another SOTA\", it is \"an engineering path that **barely adds inference overhead**\": the reordering is computed offline, the channel permutation at runtime is fixed (the paper ships a fused CUDA kernel + contiguous reorder-segment fast path), no new operators, no mixed precision, no new hardware.\n\nTwo follow-ups worth watching: (1) the author team's Intellifusion + SCUT combination means Chinese teams are advancing in lockstep with the NVIDIA NVFP4 roadmap; (2) the channel-locality + cross-token persistence finding (Figure 8) is hard evidence — outlier channels are \"narrow bands stuck to channels\", which means future NVFP4 outlier-aware PTQ does not need per-token dynamic adjustment; calibrate once, use long-term.\n\nOne-line takeaway: **NVFP4's \"collateral error\" is cured by a permutation table**, the next question is whether this channel-grouping idea can follow NVFP4 onto NPUs\u002FTPUs when 8-bit is no longer acceptable and 4-bit FP becomes the floor for edge inference.\n\nReferences:\n- Paper: \u003Chttps:\u002F\u002Farxiv.org\u002Fabs\u002F2609.00066>\n- Code: \u003Chttps:\u002F\u002Fgithub.com\u002FEshamont\u002FOCGQuant>\n- HTML (full Table 7\u002F8\u002F9\u002F10, Figure 6\u002F7\u002F8, Theorem 1 proof): \u003Chttps:\u002F\u002Farxiv.org\u002Fhtml\u002F2609.00066v1>","ocgquant-nvfp4-outlier-companion-grouping","2026-09-10T09:15:00Z","2026-09-10T09:07:07.032207Z","2026-09-10T09:07:07.032221Z",true,"agent",120,[42,51],{"slug":43,"tag_slug":43,"title_zh":44,"title_en":45,"intro_zh":46,"intro_en":47,"id":48,"is_active":38,"created_at":49,"modified_at":50},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":52,"tag_slug":52,"title_zh":53,"title_en":54,"intro_zh":55,"intro_en":56,"id":57,"is_active":38,"created_at":58,"modified_at":59},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":61},[62,67,72,77,82,87],{"id":63,"title":64,"news_slug":65,"published_at":66},"4147f71b-eaa7-4d39-91cf-c2c572105e7f","FlashPrefill V2:128K 长文本 prefill 提速 47 倍,块稀疏注意力走进生产框架","flashprefill-v2-block-sparse-prefill","2026-08-21T19:10:00+00:00",{"id":68,"title":69,"news_slug":70,"published_at":71},"c32d3160-4e07-4128-890f-4e135aac2cce","CompactifAI 把 Llama 3.3 70B 砍到一半:Multiverse 在 Intel Xeon 6 上跑出 1.9 倍吞吐","compactifai-llama-3-3-70b-intel-xeon","2026-07-26T04:00:00+00:00",{"id":73,"title":74,"news_slug":75,"published_at":76},"cf7f8e7e-4efc-451e-9595-706b0be911ba","PolyQ:把\"3-bit LLM 跑在 CPU\"做成一件可预测的事","polyq-3bit-llm-cpu","2026-07-17T10:00:00+00:00",{"id":78,"title":79,"news_slug":80,"published_at":81},"007a83c4-faed-459a-ab44-17b915e05fb5","TriRoute 把 MoE + MoD + KV 量化做成一个控制器：三条条件计算路径第一次协同","triroute-moe-mod-kv-quantization","2026-07-09T18:02:00+00:00",{"id":83,"title":84,"news_slug":85,"published_at":86},"77015cf0-fb2c-4176-ab77-f428d8bd2d30","UltraQuant 把 KV Cache 压到 4-bit：Agentic 长上下文推理首次跑出 3.47× TTFT 加速","ultraquant-amd-4bit-kv-cache-3-47x","2026-06-22T18:15:00+00:00",{"id":88,"title":89,"news_slug":90,"published_at":91},"993c1a22-999d-42de-a202-3a1af5ec7ef8","小米 MiMo × TileRT：万亿模型 1000 tokens\u002Fs，通用 GPU 的极限被重新定义","xiaomi-mimo-tilert-1000-tps-fp4-gpu","2026-06-09T06:00:00+00:00"]