[{"data":1,"prerenderedAt":-1},["ShallowReactive",2],{"news-slug-uncertainty-gate-local-llm-replay-vs-live":3,"topics-all":38,"news-related-62bc171e-2954-4c08-b3e8-d484acce7894":67},{"id":4,"title":5,"summary":6,"content":7,"original_url":8,"source_id":9,"tags":10,"translations":24,"news_slug":31,"published_at":32,"created_at":33,"modified_at":34,"is_published":35,"publish_type":36,"image_url":14,"view_count":37},"62bc171e-2954-4c08-b3e8-d484acce7894","回放AUC 0.93,实跑却分不出:本地小模型升位门控","英特尔团队研究本地小模型的精度分配:权重按8位存,解码时只取高4位,遇到不确定的解码步才升回8位重算。回放36724步里熵门控排序翻转的AUC达0.928,15%升位预算下能抓78%的翻转,但150道GSM8K实跑题上熵门控与全高精度的差距落在置信区间内,分不出高下。","本地跑大模型的人都在算同一笔账：每生成一个 token，得从内存里读多少字节的权重。英特尔（Intel Corporation）团队 11 月 10 日挂到 arXiv 上的论文换了个花招——不把模型整体压到 4 位，而是按 8 位存，每一步先用高 4 位解码，只有当这一步“看起来不确定”时，才用完整的 8 位重算一次。[arXiv:2611.14418](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2611.14418)\n\n结论先摆出来：**回放数据很漂亮，实测 150 题却分不出高下。**\n\n## 一份 4 位底子、8 位兜底的账\n\n做法比想象中朴素。Transformer 块里的每个线性权重（query、key、value、output、gate、up、down）用 round-to-nearest 量化到非对称的 8 位网格，每 64 个连续输入权重共享一个 scale 和 offset。“高版本”把 8 位全部反量化，“低版本”只保留每个码字的高 4 位，用被丢掉那一格的 midpoint 重建。两种精度读同一份存储字节，不必为低版本多存一份权重。embedding 表和输出头维持 bf16。\n\n要强调这是**模拟**：反量化后的权重就是 bf16 张量，矩阵乘也跑 bf16，没有打包 kernel，所以论文没测任何加速比，所有成本都按“每 token 读的权重字节”计算。\n\n## 回放里，这个门控几乎像作弊\n\n门控信号只有两个，都来自低版本那一步自己的输出分布：softmax 的熵，以及最大和次大 logit 的差。阈值在 32 道训练题上校准，熵阈值取 1.55、1.16、0.76、0.31 nats，margin 阈值取 0.25、0.50、1.25、2.88。\n\n回放分析把 150 道题全部 36724 步的 all-high 轨迹喂回去，在每个位置补跑一次低版本，记下熵、margin、高低分布之间的 KL，以及两个 argmax 是否一致。结果是：平均 KL 只有 0.0203 nats，3.79% 的步骤会翻转 argmax。熵在预测翻转上的 AUC 是 0.928（区间 0.924 到 0.932），margin 是 0.959（0.956 到 0.962），随机门控按构造是 0.5。\n\n在 15% 的升位预算下，熵门控抓到 78% 的翻转和 59% 的 KL 质量，margin 门控是 94% 和 55%，随机门控按期望只能抓 15%，而按真实 KL 排序的 oracle 拿到 84% 和 85%。单看这组数，门控几乎像作弊。\n\n还有一个更狠的细节：一个自由运行的 all-low 生成，通常在第 2 个 token 就跟 all-high 分道扬镳（开始偏离的位置中位数是 2），而那个偏离点的熵在全部步骤里的分位是 87%。所以门控要救的其实是开场那几个 token。\n\n## 实测 150 题：门控没站住\n\n但真正要答题的是实跑。all-high 在 150 道题上 64.0%，all-low 55.3%，差 8.7 个点（区间 -16.7 到 -0.7），McNemar 检验 p=0.053——连“全低精度更差”这件事本身在这个样本量上都只是勉强显著。\n\n实际升位率 15.7% 的熵门控（ent2）拿到 62.7%，成本是 all-high 字节数的 0.874。配对差对 all-high 是 -1.3 个点（-8.0 到 +5.3），对 all-low 是 +7.3 个点（+0.0 到 +14.7，p=0.080），区间下界正好落在 +0.0。\n\nmargin 门控 mar2 看起来更好，66.0%、字节 0.868，对 all-high 配对差 +2.0（-5.3 到 +9.3），对 all-low +10.7（+2.7 到 +18.7），p=0.017。但论文自己把这口气按住了：这是八组“门控 vs all-low”比较里的一组，没做多重比较校正，而且 mar2 是十个配置里分数最高的那个，估值本身被抬高。\n\n## 这个样本量根本分不出高下\n\n150 道配对题里，26 题是 all-high 对而 all-low 错，13 题反过来，不一致比例 0.26；论文拿这个比例当参照，算出 95% 配对差的半宽约 8.2 个点。想让一个“吃回一半差距”（4.3 个点）的门控能和 all-low 分开，半宽得压到 4.3 以下，需要大约 532 道题。回放那 36724 步虽然紧得多，但它测的是单步一致性，不产出任务准确率。\n\n## 字节账本：输出头才是大头\n\n表 2 的数字值得单独看。块权重 440.4M 参数，8 位版每次读 467.9 MB，4 位版 247.7 MB；但输出头是 155.6M 参数的 bf16，每次读 311.2 MB，和块几乎一样大。于是整个前向，all-high 是 779.1 MB，all-low 是 558.9 MB，比值 0.717，而单看块只有 0.529。头占一次高精读取的 40%、一次低精读取的 56%。\n\n更直接的后果是：升位率拉到三成的 ent3 和 mar3，实际读取的字节分别是 all-high 的 1.027 和 1.014 倍。论文说得很直白：这种门控形式下，把三分之一的步骤升位，在字节模型里无论准确率如何都是亏的。反过来看，实跑里 16% 升位率的那个门控省下了 13% 的字节——不过论文强调，这个数字还没算上任何实现成本。\n\n## 另一条路：验证式投机解码更划算\n\n同一个字节模型里，验证式投机解码（k=4）每 token 只需 all-high 的 0.845 倍字节，KL 为 0，每轮产出 4.58 个 token。熵门控在 15% 预算下是 0.867 倍字节、KL 0.0084 nats。字节数接近，零 KL 显然优于正 KL。\n\n论文的结论是有条件的：当目标模型跑得起验证、且提案接受率接近这里观测到的水平时，验证方案是更好的选择。门控的好处只在于不需要提案块和回滚，而这套字节模型把这两项成本都算成了零。\n\n## 论文自己承认的坑\n\n没跑的东西列得很清楚：实跑的随机对照组、Qwen3 思考模式的相位门控（“思考段走低、答案段走高”）、实跑 KL、1.7B 模型、第二个基准。另外权重是纯模拟的 round-to-nearest，没有校准数据、没有离群值处理——论文说，像 GPTQ、AWQ 这类调优过的 4 位方法会把高低差距缩小，门控可发挥的空间也就更小。\n\n## 所以呢\n\n这篇论文的价值在于它老老实实报了一个没结果的结论，还把字节账算得很细。对本地部署的人有两个直接的教训。第一，4 位并不是线性省：0.6B 这个尺寸上输出头跟块权重差不多大，真正的大头可能在你没量化的地方。第二，“能预测分歧”和“能提升任务准确率”是两件事——回放里 AUC 0.93 的信号，到了 150 道题的实跑里就成了噪声。\n\n如果你真打算把不确定性门控做进自己的运行时，先补上那个没跑的实时随机对照组。在门控和随机之间没有实测对照之前，字节模型里的漂亮曲线只能算假设。\n","https:\u002F\u002Farxiv.org\u002Fpdf\u002F2611.14418","7437aeb9-930c-4866-a2e9-48003c1a792b",[11,15,18,21],{"id":12,"name":13,"slug":13,"description":14,"color":14},"0ef8513a-0a26-42f0-b6f9-5b6dadded45c","efficiency",null,{"id":16,"name":17,"slug":17,"description":14,"color":14},"0a93ec8e-ea39-4693-81de-563ca8c173f7","inference",{"id":19,"name":20,"slug":20,"description":14,"color":14},"01598627-1ea6-4b27-a5d8-874971571a71","llm",{"id":22,"name":23,"slug":23,"description":14,"color":14},"b49648f9-963e-4082-8684-3d085b7358fe","quantization",[25],{"id":26,"lang":27,"title":28,"summary":29,"content":30},"f6bfc576-87db-4ceb-8c65-31b5ec781b69","en","Uncertainty-Gated Precision: 13% Fewer Bytes, No Clear Win","Intel escalated to 8 bits when uncertain: replay AUC 0.93, but 150 live GSM8K problems separated it from neither all-high nor all-low.","Every model running on a single machine spends most of its time pulling weights out of memory, so for small models decoded one token at a time the number of bytes read per token is a decent primary cost measure. A paper from Intel Corporation asks what happens if you stop picking one weight precision for the whole generation and instead allocate precision per step.\n\nThe setup is deliberately simple. Weights are stored once at 8 bits, and every decoding step runs on the top 4 bits of each code. If the low-precision output distribution looks uncertain, the step is redone with all 8 bits and that answer is kept. The uncertainty rule is cheap because it reads the low pass's own output and nothing else: either the entropy of the distribution or the gap between its top two logits. Everything reported comes from one model (Qwen3-0.6B), one benchmark (GSM8K) and one simulated quantizer, with cost counted in weight bytes read per token. [arXiv:2611.14418](https:\u002F\u002Farxiv.org\u002Fpdf\u002F2611.14418)\n\n## A 4-bit floor with an 8-bit safety net\n\nThe implementation is plainer than the framing suggests. Every linear weight in the transformer blocks is quantized once, in memory, with round-to-nearest on an asymmetric 8-bit grid, and groups of 64 consecutive input weights share a scale and an offset. The high version dequantizes all 8 bits; the low version keeps the top 4 bits of each code and reconstructs at the midpoint of the dropped cell. Both come from the same stored byte, so no extra weights are held for the low model. The embedding table and the output head stay in bf16.\n\nIt is worth flagging that this is a simulation: the dequantized weights are bf16 tensors, the matrix products run in bf16, and no packed kernel exists, so no speedup is measured. Costs are computed from how many low and high passes a configuration makes, multiplied by the bytes each pass would read.\n\n## In replay, the gate looks like cheating\n\nThe two gate signals both come from the low pass's logits. Thresholds were set on a calibration run of 32 GSM8K training problems, chosen so the low-pass signal crosses on 3, 7, 15 and 30 percent of calibration steps. The entropy thresholds are 1.55, 1.16, 0.76 and 0.31 nats; the margin thresholds are 0.25, 0.50, 1.25 and 2.88.\n\nThe replay analysis feeds the all-high trajectory of 150 problems back in and runs the low pass at every position, recording entropy, margin, the KL from the high to the low distribution, and whether the two argmax tokens agree. Over 36724 steps the mean KL is 0.0203 nats and 3.79 percent of steps flip. The area under the ROC curve for predicting a flip is 0.928 for entropy and 0.959 for the margin, with intervals of 0.924 to 0.932 and 0.956 to 0.962. A random gate gives 0.5 by construction.\n\nAt a 15 percent escalation budget the entropy gate catches 78 percent of the argmax flips and 59 percent of the KL mass, the margin gate 94 percent and 55 percent, and random escalation 15 percent of each. An oracle that ranks steps by their true KL to the high pass reaches 84 percent and 85 percent. Read on its own, the gate looks like it had solved the problem.\n\nThere is one more detail that makes the setting concrete. A free-running all-low generation typically departs from the all-high one at the second token: across the 150 problems the initial departure has a median index of 2, and the departing token sits at the 87th entropy percentile of all steps. So the steps a gate really needs to catch are concentrated right at the start.\n\n## The live run is where it falls apart\n\nThe live decode is what has to answer the question. All-high scores 64.0 percent on the 150 problems and all-low scores 55.3 percent, a gap of -8.7 points with an interval of -16.7 to -0.7 and an exact McNemar p of 0.053. Even the loss from running entirely at low precision is borderline at this sample size.\n\nThe entropy gate at a realized 15.7 percent escalation rate (ent2) scores 62.7 percent at 0.874 of the all-high weight bytes per token. Its paired difference is -1.3 points to all-high with an interval of -8.0 to +5.3, and +7.3 points to all-low with an interval of +0.0 to +14.7 at p=0.080. The margin gate mar2 looks better: 66.0 percent at 0.868 of the bytes, +2.0 points to all-high and +10.7 to all-low at p=0.017. The paper itself holds that result down, noting that it is one of eight gate-versus-all-low comparisons with no correction applied, and that mar2 is the highest-scoring of the ten configurations, which inflates its own estimate.\n\n## The sample was never going to separate them\n\nAmong the 150 paired problems, 26 were right under all-high and wrong under all-low, and 13 went the other way, so the discordant share is 0.26, which the paper uses as the reference rate. At that share, the approximate half-width of a 95 percent interval for a paired difference is about 8.2 points. Telling a gate that recovers half of the observed 8.7-point gap apart from all-low would need a half-width below 4.3 points, which at this discordant share means roughly 532 problems. The replay rests on 36724 steps and is much tighter, but it measures agreement of single steps and produces no task accuracy at all.\n\n## The output head is where the bytes are\n\nThe byte table deserves its own look. The block weights are 440.4M parameters: 467.9 MB per pass at 8 bits, 247.7 MB at 4 bits. But the output head is 155.6M parameters held in bf16 and read at 311.2 MB per pass, roughly the same size as the blocks. So a whole pass is 779.1 MB at high precision and 558.9 MB at low, a ratio of 0.717, while the block ratio alone is 0.529. The head accounts for 40 percent of a high pass and 56 percent of a low pass at this model size.\n\nThe direct consequence is that escalating too much loses money. The gates ent3 and mar3, both at roughly a 30 percent escalation rate, read 1.027 and 1.014 times the all-high bytes per token. As the paper puts it, escalating a third of all steps with this form of the gate is a loss in the byte model whatever its accuracy. By contrast the live gate at 16 percent escalation saves 13 percent of the bytes before any implementation cost.\n\n## The other road is cheaper\n\nIn the same byte model, verified speculation with k=4 needs 0.845 of the all-high bytes per token with zero KL, accepting 4.58 tokens per round. The entropy gate at a 15 percent budget needs 0.867 and leaves a mean KL of 0.0084 nats. At these byte counts the verified scheme reaches zero KL for a similar price, so it is the better choice whenever the target can be run for verification and proposals are accepted at the rate seen here. The gate's advantage is that it needs no second pass over a block of proposals, so it never pays for a rejected block, and no rollback machinery, which this cost model prices at zero.\n\n## What the paper admits it did not run\n\nThe gaps are listed plainly. No live random control was run, so the comparison against random escalation exists in replay alone, as an analytic expectation. No thinking-mode phase gate was run, even though the driver contains the modes and the design is spelled out in the paper. There is no live KL measurement, no 1.7B model and no second benchmark. The weights are simulated round-to-nearest with no calibration data and no outlier handling, and the paper notes that a tuned 4-bit method such as GPTQ or AWQ would shrink the gap between low and high, leaving any gate less room to work with.\n\n## What this actually tells you\n\nThe value here is a null result reported carefully, next to a byte accounting that is easy to get wrong. Two lessons for anyone running a small model locally. For one thing, 4-bit is not a linear saving: at the 0.6B scale the output head is about as large as the block weights, so the real bytes may be sitting in the part nobody quantized. Second, being able to predict where two precisions disagree is not the same as improving task accuracy, and a step-level AUC of 0.93 turned into noise once 150 real problems were scored.\n\nIf you are planning to put an uncertainty gate into your own runtime, run the matched live random control before anything else. Until the gate and random escalation have been compared on live data, the curve in the byte model is a hypothesis, not a result.\n","uncertainty-gate-local-llm-replay-vs-live","2026-11-10T17:57:51Z","2026-11-12T07:21:58.083256Z","2026-11-12T07:21:58.083276Z",true,"agent",213,[39,49,58],{"slug":40,"tag_slug":41,"title_zh":42,"title_en":43,"intro_zh":44,"intro_en":45,"id":46,"is_active":35,"created_at":47,"modified_at":48},"image-generation","text-to-image","图像生成：从文生图到逐框编辑的模型竞赛","Image Generation: from text-to-image to frame-level editing","文生图已是红海，战场正转向编辑与可控性：FLUX.2\u002F3 用开源和量化把生图搬进消费级显卡，Qwen-Image、Seedream 等国产模型紧咬不放，GPT Image 2 则把推理能力带进像素。本专题追踪图像生成与编辑模型的发布、架构演进、加速量化与商业化数据，看「好图」如何变成「可用的图」。","Text-to-image is a crowded race, and the frontier has moved to editing and control: FLUX.2\u002F3 push open weights and quantization onto consumer GPUs, Qwen-Image and Seedream close in from China, and GPT Image 2 brings reasoning into pixels. This topic tracks model releases, architecture shifts, acceleration work, and the business numbers behind \"pretty\" becoming \"usable\".","52de7c59-7864-4569-8ae7-3d656d26a1a1","2026-11-20T07:20:23.534506Z","2026-11-20T07:20:23.534527Z",{"slug":50,"tag_slug":50,"title_zh":51,"title_en":52,"intro_zh":53,"intro_en":54,"id":55,"is_active":35,"created_at":56,"modified_at":57},"ai-for-science","AI for Science 2026：从 UniPert 到 GPT-Rosalind 的硬核进化","AI for Science 2026: from UniPert to GPT-Rosalind","生命科学、化学材料、物理世界模型——AI 正在从\"语言工具\"变成\"实验伙伴\"。本专题收录 AI 在三大科学方向的关键节点：UniPert 统一基因与化学扰动空间、GPT-Rosalind 端到端生命科学推理、达摩院 AI 智能体 28 小时找到 4 种超导新材料、Anthropic Claude Science 把工作台做成标准品。","From language tool to lab partner — AI is reshaping life sciences, chemistry\u002Fmaterials, and physical world models. This topic covers the key milestones: UniPert unifying genetic-chemical perturbation spaces, GPT-Rosalind's end-to-end life-sciences reasoning, DAMO's AI agent discovering 4 superconducting materials in 28 hours, and Anthropic's Claude Science workbench going mainstream.","988a4300-5fab-41c4-b5d8-63711a2dc757","2026-09-10T01:34:15.296649Z","2026-09-10T01:34:15.296663Z",{"slug":59,"tag_slug":59,"title_zh":60,"title_en":61,"intro_zh":62,"intro_en":63,"id":64,"is_active":35,"created_at":65,"modified_at":66},"h3-series","MiniMax H3 系列：从开源权重到 35 倍吞吐","MiniMax H3 Series: from open weights to 35x throughput","MiniMax H3 自 2026 年 8 月开源以来节奏密集：官方把生成、参考与编辑收回一个模型；ComfyUI 当天压进 RTX 3060；摩尔线程 3 小时完成国产 GPU 适配；fal 后训练版把吞吐拉到 35 倍；FastH3 蒸馏再砍推理成本。本专题持续追踪 H3 的发布—开源—蒸馏—部署全链路。","Since MiniMax open-sourced H3 in August 2026 the pace has been relentless: one unified omni-modal model, same-day ComfyUI support down to an RTX 3060, a 3-hour Day-0 port to Moore Threads GPUs, fal's post-trained H3 Max at 35x throughput, and FastH3 distillation cutting inference cost further. This topic tracks the full H3 chain — release, open weights, distillation, deployment.","83ef0daa-3c31-4cb3-86ed-e5ee58654d5f","2026-09-08T07:33:19.942193Z","2026-09-08T07:33:19.942209Z",{"items":68},[69,74,79,84,89,94],{"id":70,"title":71,"news_slug":72,"published_at":73},"b80697dc-e9ee-40f5-a2c7-3944cd36abba","权重不等长:哈夫曼重编码,边缘LLM再省31.4%","lazyweights-huffman-edge-llm","2026-11-13T23:30:00+00:00",{"id":75,"title":76,"news_slug":77,"published_at":78},"081148f2-39eb-432d-8841-5cb1e58ed527","省内存还是多跑几遍:本地小模型怎么花推理预算才划算","quantization-bits-vs-option-order-budget","2026-11-12T09:10:00+00:00",{"id":80,"title":81,"news_slug":82,"published_at":83},"a6f5abcc-f1eb-4b6c-ba50-062836f50d0d","量化前先算一遍:二阶筛查排序能信,绝对值不能","quantization-fisher-screen-pre-registered-audit","2026-11-11T03:30:00+00:00",{"id":85,"title":86,"news_slug":87,"published_at":88},"93963f65-e9a2-4058-ba37-563154d1aebd","bf16 状态:推理模型准确率 85.6 掉到 70.8","bf16-state-cache-hybrid-reasoning","2026-11-10T07:15:00+00:00",{"id":90,"title":91,"news_slug":92,"published_at":93},"8c0d584e-4bb2-4d00-a5cd-9529824f756d","量化推理模型先管开头30步:CodeQuant2","codequant2-reasoning-model-quantization","2026-11-09T17:10:11+00:00",{"id":95,"title":96,"news_slug":97,"published_at":98},"8b9f54f1-334a-4725-9c50-43dd1f018fba","TWIG:把权重存成一棵树,量化推理全打通","twig-weight-tree-llm-quantization","2026-10-27T01:30:00+00:00"]