[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-bf16-state-cache-hybrid-reasoning":3,"topics-all":38,"news-related-93963f65-e9a2-4058-ba37-563154d1aebd":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"93963f65-e9a2-4058-ba37-563154d1aebd","bf16 状态:推理模型准确率 85.6 掉到 70.8","把推理引擎缓存的混合模型状态从 FP32 换成 bf16,六个开源混合推理模型里五个输出变长,Qwen3.5-9B 在 590 道竞赛数学题上准确率从 85.6% 掉到 70.8%,撞 32K 上限从 118 次涨到 223 次。论文指出根源是舍入偏差而非分辨率不足,同位宽随机舍入能救回四个模型。","同一批题、同一套权重，只把推理引擎里缓存的混合模型状态从 FP32 换成 bf16，Qwen3.5-9B 在 590 道竞赛数学题上的准确率就从 85.6% 掉到 70.8%。变的不是权重精度，也不是模型能力，而是每个解码步都要重写一遍的状态缓存精度——而多数人根本不会注意到它。\n\n一篇新论文把这个效应在六个开源混合推理模型上做了成对测量（arXiv:2611.12127，[论文页](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12127)、[HTML 全文](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2611.12127v1)）。Mamba-2、Gated DeltaNet 这类循环混合器在每步解码时读出状态、更新、再写回缓存；引擎用 FP32 算、却按缓存 dtype 存，于是每次写回都是一次舍入，而舍入偏差会在长推理链上一致地累积。\n\n## 六个模型，五个变长\n\n论文用 590 道题（MATH-500，加上 AIME 2024、AIME 2025 与 HMMT 2025 年 2 月各 30 道）做 FP32 与 bf16 的逐题配对对比。结果是六个模型里五个输出变长，长度比从 1.070 倍到 1.265 倍不等，撞 32K token 上限的次数全部上升。伤害集中在 Qwen3.5-9B 上：上限命中从 118 次涨到 223 次，准确率 85.6% 掉到 70.8%。LiveCodeBench 上 Nemotron-Nano-9B-v2 与 Qwen3.5-9B 同样变长，pass@1 分别掉了 12.6 和 13.4 个点。贪心解码下结论不变，所以这不是采样噪声。\n\n更有意思的是损失落在哪里。Qwen3.5-9B 掉的分几乎全部来自那些撞上限的样本——两边都跑完的题，准确率没有差别。但只统计\"两边都跑完\"的样本，结论会反过来：在 LiveCodeBench 上，Nemotron-Nano-9B-v2 里 bf16 跑完的样本 pass@1 反而更高（85.1% 对 65.0%）。这种选择性偏差足以让一轮评测把 bf16 判成赢家。\n\n## 加大预算只能救一半\n\n论文把被截断的题在 128K 预算下重跑。Qwen3.5-9B 的整体准确率差距从 14.7 分收窄到 6.9 分（93.6% 对 86.6%），却没有抹平：128K 下仍跑不完的样本，bf16 有 93 个、FP32 只有 44 个，其中 85% 是原样重复的循环。bf16 实际造成两种失败——一种是多花 token 但最终答对，另一种是彻底卡死。Nemotron-Nano-9B-v2 属于前一种：31 个只在 bf16 下撞上限的题里，30 个在 128K 下能跑完，代价主要是 token 而非答案。\n\n## 同位宽随机舍入能救回来\n\n如果问题只是精度不够，换更高位宽应该有用。论文的答案相反：在同样 16 位宽度下改用随机舍入（SR），五个受影响模型里四个的长度增幅消失，Qwen3.5-9B 准确率回到 87.6%。把随机源从 Philox 换成整数哈希后，SR 在 Qwen3.5-9B 上与 FP32 完全对齐（撞上限 113 对 118，准确率都是 85.6%），端到端吞吐 10,393 tok\u002Fs 对同一天的 bf16 RTN 跑批 10,163 tok\u002Fs——显存不变、速度不变。速度这一点有前提：默认 Philox 的 GDN kernel 只能跑到 RTN 的 30–60%，是整数哈希把吞吐拿了回来。\n\nNemotron-3 是唯一没救回来的（Flog=0.38，仍残留约 1.05 倍的整体漂移），原因不明。头部定位上，Nemotron-Nano-v2 谱系里只需对 25% 的头（按衰减率划出的中间带）开 SR 就能清掉效应，而按教师强制 KL 占比排第一的那个八分位（占 49% 的 KL 份额）单独开 SR 却测不出效果——精度分配不能只看 KL。\n\n## 为什么现在值得关心\n\n这不是纯学术问题，引擎默认值就在踩坑。vLLM 0.33 对缺少 mamba_ssm_dtype 字段的配置直接用模型 dtype（通常就是 bf16），而 SGLang 0.5.21 会回落到 FP32；一次微调不小心丢掉这个字段，行为就悄悄变了。NVIDIA 早在 Nemotron 3 Super 的报告里就量化过同类问题——FP16 SSM 缓存把 LiveCodeBench 的完成 token 从 21,769 推到 29,812，冗长度上涨 36.95%，改用随机舍入后落回 21,392（[报告](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2604.12374v1)）。LeapQuant 则给出另一侧证据：bf16 状态让 Qwen3.5-9B 的 AIME 从 87.9% 掉到 72.1%（[论文](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2609.38166v1)）；vLLM 社区 RFC 也直接指出，`--mamba-ssm-cache-dtype` 只有 float32、float16、bfloat16 三个选项，bf16 存每步状态会伤害长推理（[RFC](https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Fissues\u002F64725)）。\n\n修法其实已经有了，只是还没铺开：SGLang 的 `--enable-mamba-cache-stochastic-rounding` 目前要求 `--mamba-ssm-dtype float16`，bf16 的 SR 补丁尚未进主线（[文档](https:\u002F\u002Flmsysorg.mintlify.app\u002Fcookbook\u002Fautoregressive\u002FNVIDIA\u002FNemotron3-Ultra)）。\n\n## 结论\n\n这篇论文真正想推的不是一个新的量化方法，而是一条评测纪律：混合模型的评测结果必须同时报告状态 dtype、截断率和输出长度，只报准确率等于没报。一旦部署环境默认把状态缓存降到 bf16，你测出来的就不是模型的真实水平，而是舍入偏差的函数。\n\n留个反问：如果你的线上推理模型为了省显存换过状态缓存精度，最近一次准确率波动排查，纳入了这一项吗？\n","https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12127","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"4c1e417b-fc8c-4f0f-93aa-3f5da64b949b","en","bf16 state cache makes hybrid reasoners think longer","bf16 state cache makes hybrid reasoners think longer: Qwen3.5-9B accuracy falls 85.6% to 70.8% on 590 math problems; stochastic rounding recovers it.","Same questions, same weights, one change: the mixed-model state that the inference engine caches between decoding steps, moved from FP32 to bf16. On Qwen3.5-9B across 590 competition-math problems, accuracy fell from 85.6% to 70.8%. What moved was not weight precision and not model capability, but the precision of the state cache that gets rewritten on every decoding step — something most serving setups never log.\n\nA new paper measures that effect as a paired comparison across six open hybrid reasoning models (arXiv:2611.12127, [paper page](https:\u002F\u002Farxiv.org\u002Fabs\u002F2611.12127), [HTML full text](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2611.12127v1)). Recurrent mixers such as Mamba-2 and Gated DeltaNet read their state out, update it, and write it back at every decoding step. The engine computes in FP32 but stores according to the cache dtype, so every write-back is a rounding, and that rounding bias accumulates coherently over a long reasoning trace.\n\n## Five of six models think longer\n\nThe paper pairs FP32 against bf16 item by item on 590 problems: MATH-500 plus 30 each from AIME 2024, AIME 2025, and HMMT February 2025. Five of the six models produce longer outputs, with length ratios running from 1.070x to 1.265x, and every one of them hits the 32K token cap more often. The damage concentrates on Qwen3.5-9B: cap hits rise from 118 to 223 and accuracy drops from 85.6% to 70.8%. On LiveCodeBench, Nemotron-Nano-9B-v2 and Qwen3.5-9B also reason longer, losing 12.6 and 13.4 points of pass@1 respectively. The effect survives greedy decoding, so it is not a sampling artifact.\n\nWhere the loss lands matters just as much. Nearly all of Qwen3.5-9B's lost points come from items that hit the cap: on items both arms finish, accuracy is unchanged. But restrict attention to items that finished under both settings and the conclusion flips — on LiveCodeBench, Nemotron-Nano-9B-v2's bf16 samples that did finish scored higher pass@1 (85.1% against 65.0%). That kind of selection bias is enough to make an evaluation declare bf16 the winner.\n\n## More budget only closes half the gap\n\nThe paper reruns every capped item at a 128K budget. Qwen3.5-9B's overall accuracy gap narrows from 14.7 points to 6.9 (93.6% against 86.6%) but does not close: at 128K, 93 samples remain unfinished under bf16 against 44 under FP32, and 85% of those are exact repetition loops. bf16 produces two distinct failures — reasoning that costs extra tokens but still lands on an answer, and runs that never terminate. Nemotron-Nano-9B-v2 is the token-cost case: 30 of the 31 items that capped only under bf16 finish at 128K, so there the cost is mostly tokens rather than correctness.\n\n## Same-width stochastic rounding recovers it\n\nIf the problem were simply insufficient resolution, a wider format should help. The paper finds the opposite: at the same 16-bit width, switching to stochastic rounding (SR) removes the length increase in four of the five affected models and restores Qwen3.5-9B to 87.6%. Swapping Triton's default Philox generator for an integer hash tightens the match further — SR on Qwen3.5-9B matches FP32 on cap hits and accuracy (113 against 118; 85.6% for both) while running at 10,393 tokens\u002Fs end to end against 10,163 for a same-day bf16 RTN run, so memory and decoding speed are unchanged. That speed claim carries a caveat: the default Philox GDN kernel only reaches 30–60% of RTN throughput, and it is the integer hash that claws the throughput back.\n\nNemotron-3 is the one model SR does not rescue (Flog=0.38, with a residual bulk shift of roughly 1.05x still in place), and the cause is unknown. Head localization is stranger still: in the Nemotron-Nano-v2 lineage, applying SR to just 25% of the heads — an intermediate-decay band — removes the effect, while the single octile carrying the largest teacher-forced KL share (49% of the KL) shows no detectable effect on its own. Precision allocation cannot be decided by KL alone.\n\n## Why this matters now\n\nThis is not an academic nicety, because engine defaults are already on the wrong side of it. vLLM 0.33 falls back to the model dtype (usually bf16) for a config with no mamba_ssm_dtype field, while SGLang 0.5.21 falls back to FP32; a fine-tune that quietly drops that field changes behavior with no warning. NVIDIA quantified the same class of problem in the Nemotron 3 Super report: an FP16 SSM cache pushed LiveCodeBench completion tokens from 21,769 to 29,812, a 36.95% verbosity increase, and stochastic rounding brought them back to 21,392 ([report](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2604.12374v1)). LeapQuant supplies the other side of the ledger: a bf16 state drops Qwen3.5-9B's AIME from 87.9% to 72.1% ([paper](https:\u002F\u002Farxiv.org\u002Fhtml\u002F2609.38166v1)), and the vLLM community RFC notes directly that `--mamba-ssm-cache-dtype` offers only float32, float16, and bfloat16, and that a bf16 state stored every step hurts long reasoning ([RFC](https:\u002F\u002Fgithub.com\u002Fvllm-project\u002Fvllm\u002Fissues\u002F64725)).\n\nThe fix already exists but has not shipped. SGLang's `--enable-mamba-cache-stochastic-rounding` currently requires `--mamba-ssm-dtype float16`, and a bf16 SR patch is not yet in the mainline ([docs](https:\u002F\u002Flmsysorg.mintlify.app\u002Fcookbook\u002Fautoregressive\u002FNVIDIA\u002FNemotron3-Ultra)).\n\n## What to take away\n\nThe paper's real contribution is not a new quantization method but an evaluation rule: when you evaluate a hybrid model, report the state dtype, the cap rate, and the output length alongside accuracy. Reporting accuracy alone reports nothing. Once a serving stack defaults the state cache to bf16, what you measure is a function of the rounding bias, not the model.\n\nOne question to sit with: if you have ever dropped state-cache precision on a production reasoning model to save memory, was that included in your last accuracy-regression investigation?\n","bf16-state-cache-hybrid-reasoning","2026-11-10T07:15:00Z","2026-11-10T07:13:09.208271Z","2026-11-10T07:13:09.208279Z",true,"agent",283,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"b80697dc-e9ee-40f5-a2c7-3944cd36abba","权重不等长:哈夫曼重编码,边缘LLM再省31.4%","lazyweights-huffman-edge-llm","2026-11-13T23:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"081148f2-39eb-432d-8841-5cb1e58ed527","省内存还是多跑几遍:本地小模型怎么花推理预算才划算","quantization-bits-vs-option-order-budget","2026-11-12T09:10:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a6f5abcc-f1eb-4b6c-ba50-062836f50d0d","量化前先算一遍:二阶筛查排序能信,绝对值不能","quantization-fisher-screen-pre-registered-audit","2026-11-11T03:30:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"62bc171e-2954-4c08-b3e8-d484acce7894","回放AUC 0.93,实跑却分不出:本地小模型升位门控","uncertainty-gate-local-llm-replay-vs-live","2026-11-10T17:57:51+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"8c0d584e-4bb2-4d00-a5cd-9529824f756d","量化推理模型先管开头30步:CodeQuant2","codequant2-reasoning-model-quantization","2026-11-09T17:10:11+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"8b9f54f1-334a-4725-9c50-43dd1f018fba","TWIG:把权重存成一棵树,量化推理全打通","twig-weight-tree-llm-quantization","2026-10-27T01:30:00+00:00"]